Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

The costliest mistake in in-house LLM investments is buying GPUs based on a wrong estimate of demand. The programme works step by step through calculating weight, KV cache and activation memory, deriving concurrency and tokens/s requirements from user numbers and request profiles, and how TTFT and latency SLOs drive hardware choice.

Participants compare options such as NVIDIA H100, H200, B200, L40S and AMD MI300X by memory bandwidth, NVLink and power draw, then use an inference benchmark based on real measurements to prepare a capacity plan and an on-prem versus cloud cost comparison for their own scenario.

Who Should Attend?

  • Platform and infrastructure architects contributing to GPU purchase decisions
  • AI engineers building an in-house LLM platform
  • Technical managers responsible for IT budget and capacity
  • Prerequisite: basic LLM inference concepts

Key Takeaways

  • Calculate VRAM and GPU count for a model and workload
  • Derive concurrency and throughput targets from user profiles
  • Compare GPU models by bandwidth, memory and cost
  • Validate the capacity plan with benchmark data
  • Run a total cost analysis for on-prem and cloud GPU options

Curriculum

Memory Math
  • Formulas for weight, KV cache and activation memory
  • Impact of context length, batch size and quantization
Workload Modelling
  • User count, request rate and input/output token profile
  • TTFT, inter-token latency and throughput SLOs
GPU and Server Options
  • Comparing H100, H200, B200, L40S and MI300X
  • NVLink, PCIe, networking, power and cooling requirements
Validating with Benchmarks
  • Inference benchmarks with realistic load scenarios
  • Translating results into GPU count
Cost and Growth Plan
  • Total cost of on-prem, colocation and cloud GPUs
  • Growth scenarios and capacity plan document

Other Courses in This Category

  • In-House LLM Platform Architecture: From Hardware Sizing to Air-Gapped Production

    End-to-end in-house LLM platform design for data residency and regulated environments: GPU sizing, Kubernetes cluster, vLLM/Triton serving, model gateway, security and air-gapped production architecture. Built for platform and infrastructure teams.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Air-Gapped LLM Deployment, Offline Registry Mirroring and Update Operations

    Deploying and updating LLM services in air-gapped networks: offline container, PyPI and Hugging Face mirroring, safetensors and SBOM verification, GPU driver bundling and zero-downtime patching. Built for security-focused platform teams.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Running Local LLMs on Desktop and Workstations: llama.cpp, GGUF and LM Studio

    Running local LLMs on desktops and workstations with llama.cpp, GGUF and LM Studio: k-quant selection, CPU+GPU layer offloading, Metal/CUDA/Vulkan acceleration and local OpenAI-compatible APIs. For developers and analysts.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Self-Hosted AI Coding Assistants with Local LLMs

    Building self-hosted AI coding assistants without sending source code outside the company: Qwen2.5-Coder and DeepSeek-Coder models, FIM autocomplete, Continue.dev/Tabby IDE plugins, vLLM serving and codebase RAG integration.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Open-Weight LLM Selection, Licensing and Benchmarking for Enterprise Use

    Selecting open-weight LLMs for enterprise use: Llama, Qwen, Mistral, Gemma and DeepSeek families; commercial licence analysis, dense vs MoE trade-offs, Turkish and tool-calling capability, and building a benchmark matrix on company data.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Enterprise Model Serving with NVIDIA Triton Inference Server and KServe

    Multi-model GPU serving with NVIDIA Triton Inference Server and KServe: TensorRT-LLM, ONNX and Python backends, dynamic batching, model ensembles, KServe InferenceService, Knative autoscaling and tuning with Model Analyzer.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View

Get in touch