Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Maintaining separate serving stacks for tabular XGBoost models, PyTorch/TensorFlow deep learning networks and custom Python libraries across different data science teams creates severe operational sprawl. This course teaches how to consolidate all of these frameworks behind a single standard API and runtime using Triton's multi-backend architecture.

Participants deploy models on the LibTorch, TensorFlow, ONNX Runtime, TensorRT, OpenVINO, Forest Inference Library (FIL) and Python backends, isolate custom Python environments with conda-pack and wire up MLflow-to-Triton deployments. Teams leave with a unified reference repository and a backend selection guide for their model portfolio.

Who Should Attend?

  • MLOps and platform engineers supporting multiple ML/DL frameworks in production
  • ML engineers combining tabular (XGBoost/LightGBM), vision and NLP models in one service
  • AI developers migrating custom Python libraries into Triton
  • Prerequisite: Triton Fundamentals knowledge and experience with at least one ML framework

Key Takeaways

  • Select between PyTorch, TensorFlow, ONNX, TensorRT, OpenVINO and FIL backends per workload
  • Accelerate XGBoost, LightGBM and scikit-learn tree models on GPU/CPU with the FIL backend
  • Package custom-dependency Python models using TritonPythonModel and conda-pack
  • Resolve tensor naming, data-type and device (CPU/GPU) mismatches across backends
  • Deploy multi-framework models to Triton from MLflow Model Registry and KServe

Curriculum

Triton Backend Ecosystem and Selection Criteria
  • Backend architecture, shared libraries and building custom Triton containers (build.py)
  • Backend decision matrix by hardware target and model family
Deep Learning Backends: PyTorch, TensorFlow, ONNX and TensorRT
  • Integration nuances for TorchScript (pytorch_libtorch) and SavedModel (tensorflow)
  • Configuring ONNX Runtime, TensorRT plans and the OpenVINO backend for CPUs
Tabular and Classical ML Models with the FIL Backend
  • Serving XGBoost, LightGBM and scikit-learn Random Forest models via FIL
  • Treelite serialization, CPU/GPU crossover thresholds and high-QPS tabular scoring
Python Backend and Custom Environment Packaging
  • Running spaCy, Hugging Face or custom logic with TritonPythonModel (initialize, execute, finalize)
  • Isolating per-model Python dependencies with conda-pack and EXECUTION_ENV_PATH
Hybrid Pipelines and MLflow/KServe Integration
  • Combining Python preprocessing, ONNX/TensorRT and FIL models on a single server
  • Lifecycle management with the mlflow-triton plugin and KServe multi-model serving

Other Courses in This Category

  • Serving Machine Learning and LLM Models with FastAPI

    Building production-ready ML and LLM inference services with FastAPI: model loading via lifespan, Pydantic v2 schemas, async/threadpool execution, SSE streaming, Gunicorn/Uvicorn tuning and Prometheus metrics. For developers and ML engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Serving Machine Learning Models with Flask

    Building lightweight ML prediction services with Flask and WSGI: Application Factory, Blueprints, Marshmallow/Pydantic validation, Gunicorn worker/thread tuning and Docker packaging. For data scientists and Python developers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Distributed Model and LLM Serving with Ray Serve

    Python-native distributed model and LLM serving with Ray Serve: DeploymentGraph composition, fractional GPU sharing, dynamic batching, replica autoscaling, vLLM integration and KubeRay. For ML and platform engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Model Versioning, Rollout Strategies and Safe Replacement

    Updating production models safely without downtime: MLflow Model Registry, semantic versioning, shadow deployments, canary rollouts, blue/green switches, A/B testing and automated rollback. For MLOps and ML engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Scaling Deployed Machine Learning and LLM Services

    Horizontal and vertical scaling for live model services: autoscaling on queue/GPU metrics with Kubernetes HPA, KEDA and Knative, reducing cold starts, MIG/time-slicing and load balancing. For MLOps, SRE and platform engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Introduction to Model Deployment

    The core steps for moving a trained model from notebook to production: serialization, serving via an API, container packaging, basic monitoring and rollback. For data scientists and developers new to model deployment.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View

Get in touch