Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Text Generation Inference (TGI) is a serving solution built to turn models on the Hugging Face Hub into production endpoints quickly. The programme covers TGI's router/launcher architecture, Docker installation, multi-GPU sharding, bitsandbytes/AWQ/GPTQ/FP8 quantization options and OpenAI-compatible use through the Messages API.

Exercises configure token access to gated models, JSON-schema output with guidance, tool calling and Prometheus metrics. Comparing TGI with vLLM and SGLang, participants prepare a reasoned assessment of which engine fits their organisation's existing stack.

Who Should Attend?

  • ML engineers using Hugging Face Hub and Transformers
  • Platform teams building model serving infrastructure
  • Developers using TGI on Inference Endpoints or SageMaker
  • Prerequisite: Docker and Python

Key Takeaways

  • Run TGI with Docker on single or multiple GPUs
  • Tune quantization and token limits to the hardware
  • Integrate via the Messages API, streaming and structured output
  • Monitor TGI metrics and run load tests
  • Make a reasoned choice between TGI, vLLM and SGLang

Curriculum

TGI Architecture and Ecosystem
  • Router, launcher and model shards
  • Hugging Face Hub, gated models and licences
Installation and Configuration
  • Launching with Docker, sharding and token limits
  • bitsandbytes, AWQ, GPTQ and FP8 quantization options
Using the API
  • Messages API, streaming and chat templates
  • JSON output with guidance and tool calling
Monitoring and Scaling
  • Prometheus metrics and OpenTelemetry tracing
  • Running on Inference Endpoints and Kubernetes
Engine Comparison
  • Benchmarking TGI, vLLM and SGLang on the same model
  • Selection criteria for the organisation's stack

Other Courses in This Category

  • Running Local LLMs with Ollama

    Running open-weight models on laptops, workstations or servers with Ollama: model library, Modelfile, REST/OpenAI-compatible API, Open WebUI, embeddings and a shared team setup. For fast prototyping and confidential-data scenarios.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • LLM Serving Stack Mastery: FastAPI, vLLM, KServe and Triton

    Combines a FastAPI model API layer, vLLM for LLM inference, KServe for model management on Kubernetes and Triton for multi-framework serving in one architecture. Participants build an end-to-end, scalable model serving stack.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    6 modules
    View
  • LLM Quantization: INT8, INT4, FP8, FP16 and BF16 in Practice

    FP16/BF16, FP8, INT8 and INT4 precision options for LLMs; post-training quantization with GPTQ, AWQ, GGUF and bitsandbytes, measuring quality loss with evals and verifying VRAM and throughput gains through benchmarks.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Dynamic Quantization for Neural Networks and LLMs

    Accelerating models with dynamic quantization without calibration data: PyTorch and ONNX Runtime dynamic quantization, FP8 dynamic activation scaling, comparison with static methods and measurement on CPU and GPU.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Quantization-Aware Training, QLoRA Fine-Tuning and Quantized Serving

    Covers quantization end to end from training to serving: 4-bit fine-tuning with QLoRA, quantization-aware training (QAT), merging adapters, re-quantizing and serving in vLLM. For teams adapting models to company data on a single GPU.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • GPU Capacity Planning and Hardware Sizing for LLM Inference

    GPU capacity planning for LLM workloads: deriving VRAM, GPU count and server needs from model size, context length, concurrent users and SLO targets, plus a cost comparison of H100/H200/B200, L40S and cloud options.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View

Get in touch