Model Online / On-Site
Duration
Modules 6 modules

Course Objective & Scope

In enterprise environments a single serving tool is rarely enough: LLMs, embedding and reranker models, classic ML models and business rules are served on different layers. This advanced programme shows how to position a FastAPI API layer with authentication, rate limiting and business logic in front of vLLM, Triton Inference Server and KServe InferenceService resources.

In the labs participants build a KServe service with a vLLM runtime on Kubernetes, an embedding/ONNX model served by Triton and an asynchronous FastAPI gateway in front, then validate the architecture with autoscaling, canary rollout and a cost/performance comparison.

Who Should Attend?

  • Senior ML and platform engineers designing a model serving platform
  • MLOps teams operating multiple model types in production
  • AI solution architects
  • Prerequisite: Python, Docker, Kubernetes basics and LLM serving experience

Key Takeaways

  • Build asynchronous, streaming-capable model APIs with FastAPI
  • Run vLLM as a KServe runtime on Kubernetes
  • Serve ONNX, TensorRT and Python backend models with Triton
  • Apply autoscaling, canary and traffic-splitting strategies
  • Justify which component fits which workload

Curriculum

Serving Stack Architecture
  • Separating API layer, inference engine and orchestration
  • Reference architecture for LLM, embedding and classic ML models
Model APIs with FastAPI
  • Async endpoints, SSE streaming and Pydantic schemas
  • Authentication, rate limiting and error handling
vLLM Integration
  • Connecting an OpenAI-compatible vLLM service to the API layer
  • Connection pooling, timeouts and backpressure
Triton Inference Server
  • Model repository, ONNX/TensorRT/Python backends
  • Dynamic batching and model ensembles
Serving on Kubernetes with KServe
  • InferenceService, ServingRuntime and vLLM/Triton runtimes
  • Autoscaling, canary rollout and traffic splitting
End-to-End Validation
  • Load testing and cost/performance comparison
  • Target serving architecture draft for the organisation

Other Courses in This Category

  • LLM Quantization: INT8, INT4, FP8, FP16 and BF16 in Practice

    FP16/BF16, FP8, INT8 and INT4 precision options for LLMs; post-training quantization with GPTQ, AWQ, GGUF and bitsandbytes, measuring quality loss with evals and verifying VRAM and throughput gains through benchmarks.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Dynamic Quantization for Neural Networks and LLMs

    Accelerating models with dynamic quantization without calibration data: PyTorch and ONNX Runtime dynamic quantization, FP8 dynamic activation scaling, comparison with static methods and measurement on CPU and GPU.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Quantization-Aware Training, QLoRA Fine-Tuning and Quantized Serving

    Covers quantization end to end from training to serving: 4-bit fine-tuning with QLoRA, quantization-aware training (QAT), merging adapters, re-quantizing and serving in vLLM. For teams adapting models to company data on a single GPU.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • GPU Capacity Planning and Hardware Sizing for LLM Inference

    GPU capacity planning for LLM workloads: deriving VRAM, GPU count and server needs from model size, context length, concurrent users and SLO targets, plus a cost comparison of H100/H200/B200, L40S and cloud options.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • In-House LLM Platform Architecture: From Hardware Sizing to Air-Gapped Production

    End-to-end in-house LLM platform design for data residency and regulated environments: GPU sizing, Kubernetes cluster, vLLM/Triton serving, model gateway, security and air-gapped production architecture. Built for platform and infrastructure teams.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Air-Gapped LLM Deployment, Offline Registry Mirroring and Update Operations

    Deploying and updating LLM services in air-gapped networks: offline container, PyPI and Hugging Face mirroring, safetensors and SBOM verification, GPU driver bundling and zero-downtime patching. Built for security-focused platform teams.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View

Get in touch