Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

ONNX and ONNX Runtime provide the standard path for running models trained across PyTorch, TensorFlow, scikit-learn and XGBoost inside a single lightweight runtime with hardware acceleration. This course covers the practical workflow from exporting models into an ONNX computation graph to running high-speed inference in production.

Participants convert models using torch.onnx, tf2onnx, skl2onnx and Hugging Face Optimum, inspect graphs with Netron, verify numerical equivalence and apply ONNX Runtime graph optimisations and INT8 quantization. They finish with a reusable conversion and validation script built on one of the organisation's models.

Who Should Attend?

  • ML engineers seeking low-latency serving with minimal runtime dependencies
  • MLOps teams standardising inference across multiple training frameworks
  • Software developers integrating models into non-Python environments (C++, C#, Java, edge/browser)
  • Prerequisite: Python and working knowledge of PyTorch, TensorFlow or scikit-learn

Key Takeaways

  • Export PyTorch, TensorFlow, scikit-learn and Hugging Face models to ONNX with dynamic axes
  • Manage ONNX opset versions and inspect operator graphs using Netron
  • Automate numerical equivalence checks between original and ONNX model outputs
  • Configure ONNX Runtime Execution Providers (CPU, CUDA, TensorRT, OpenVINO)
  • Reduce model size and latency using graph optimisations and FP16/INT8 quantization

Curriculum

The ONNX Ecosystem and Graph Structure
  • Intermediate Representation (IR), operators, tensors and opset versions
  • Visualising and validating model graphs with Netron and onnx.checker
Converting Models from Frameworks to ONNX
  • Exporting from PyTorch with torch.onnx.export and defining dynamic_axes
  • Converting models with tf2onnx, skl2onnx and Hugging Face Optimum
Numerical Validation and Troubleshooting Exports
  • Checking output tolerances with numpy.testing.assert_allclose
  • Resolving unsupported operators, frozen shapes and control-flow pitfalls
ONNX Runtime and Execution Providers
  • InferenceSession, SessionOptions, thread settings and I/O binding
  • Selecting CPU, CUDA, TensorRT and OpenVINO Execution Providers
Graph Optimisation and Quantization
  • Constant folding, operator fusion (ORT_ENABLE_ALL) and saving offline-optimised graphs
  • Dynamic and static INT8 quantization with speed-versus-accuracy benchmarks

Other Courses in This Category

  • NVIDIA Triton Inference Server Fundamentals

    High-performance model serving with NVIDIA Triton Inference Server: Model Repository layout, config.pbtxt, dynamic batching, concurrent model execution, HTTP/gRPC clients and Perf Analyzer. For ML and MLOps engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Advanced NVIDIA Triton: Ensembles, BLS, LLMs and Tuning

    Automated tuning with Triton Model Analyzer, Ensembles and Business Logic Scripting (BLS), stateful/sequence batching, shared memory (IPC) and LLM serving with TensorRT-LLM/vLLM backends. For senior ML engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Integrating Multi-Framework Models with NVIDIA Triton

    Unifying PyTorch, TensorFlow, ONNX, TensorRT, OpenVINO and scikit-learn/XGBoost (FIL) models alongside custom Python backends on a single Triton server. For ML and MLOps engineers managing heterogeneous model portfolios.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Serving Machine Learning and LLM Models with FastAPI

    Building production-ready ML and LLM inference services with FastAPI: model loading via lifespan, Pydantic v2 schemas, async/threadpool execution, SSE streaming, Gunicorn/Uvicorn tuning and Prometheus metrics. For developers and ML engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Serving Machine Learning Models with Flask

    Building lightweight ML prediction services with Flask and WSGI: Application Factory, Blueprints, Marshmallow/Pydantic validation, Gunicorn worker/thread tuning and Docker packaging. For data scientists and Python developers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Distributed Model and LLM Serving with Ray Serve

    Python-native distributed model and LLM serving with Ray Serve: DeploymentGraph composition, fractional GPU sharing, dynamic batching, replica autoscaling, vLLM integration and KubeRay. For ML and platform engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View

Get in touch