Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Running a model from Hugging Face in a notebook is very different from serving it as a measurable, multi-user service. This programme explains in plain terms how model weights, tokenizer, context window and VRAM relate, what happens in the prefill and decode phases of inference, and which serving options such as Ollama, vLLM and TGI suit which scenario.

In hands-on labs participants stand up a Llama, Qwen or Gemma model as an OpenAI-compatible endpoint, run a simple inference benchmark and draft an initial deployment plan suited to their organisation.

Who Should Attend?

  • Software developers starting to run LLMs on their own infrastructure
  • Platform, DevOps and systems engineers
  • Data scientists who select models and want to understand serving
  • Prerequisite: basic Linux command line and Python

Key Takeaways

  • Estimate required VRAM from a model's parameter count and precision
  • Serve an open-weight model as an OpenAI-compatible API
  • Choose between Ollama, vLLM and TGI for a given scenario
  • Measure time to first token, tokens/s and concurrency
  • Prepare a first model deployment draft for the organisation

Curriculum

How LLM Inference Works
  • Tokenizer, prefill and decode phases
  • Context window, KV cache and memory consumption
Hardware and Model Size
  • VRAM math for parameter count, FP16/BF16 and quantized weights
  • Single-GPU, multi-GPU and CPU options
Overview of Serving Engines
  • Comparing Ollama, llama.cpp, vLLM, TGI and SGLang
  • OpenAI-compatible API, streaming and chat templates
Standing Up the First Endpoint
  • Starting a model service with Docker and core parameters
  • Calling the API from an application
Measurement and Deployment Draft
  • TTFT, tokens/s and concurrent request tests
  • Organisation-specific deployment checklist

Other Courses in This Category

  • LLM Serving with vLLM: Production Inference Deployment

    High-throughput LLM serving with vLLM: PagedAttention, continuous batching, tensor parallelism, prefix caching, multi-LoRA serving and Prometheus metrics. Teams leave with a production-ready vLLM deployment and benchmark report.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Hugging Face Text Generation Inference (TGI) for LLM Serving

    Serving LLMs with Hugging Face Text Generation Inference (TGI): Docker setup, sharding, quantization, Messages API, structured output with guidance and Inference Endpoints. A hands-on programme for teams in the Hugging Face ecosystem.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Running Local LLMs with Ollama

    Running open-weight models on laptops, workstations or servers with Ollama: model library, Modelfile, REST/OpenAI-compatible API, Open WebUI, embeddings and a shared team setup. For fast prototyping and confidential-data scenarios.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • LLM Serving Stack Mastery: FastAPI, vLLM, KServe and Triton

    Combines a FastAPI model API layer, vLLM for LLM inference, KServe for model management on Kubernetes and Triton for multi-framework serving in one architecture. Participants build an end-to-end, scalable model serving stack.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    6 modules
    View
  • LLM Quantization: INT8, INT4, FP8, FP16 and BF16 in Practice

    FP16/BF16, FP8, INT8 and INT4 precision options for LLMs; post-training quantization with GPTQ, AWQ, GGUF and bitsandbytes, measuring quality loss with evals and verifying VRAM and throughput gains through benchmarks.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Dynamic Quantization for Neural Networks and LLMs

    Accelerating models with dynamic quantization without calibration data: PyTorch and ONNX Runtime dynamic quantization, FP8 dynamic activation scaling, comparison with static methods and measurement on CPU and GPU.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View

Get in touch