Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Capable open-weight models can run on developer laptops and workstations without data-centre GPUs. This course provides hands-on training in the llama.cpp architecture, the GGUF file format, k-quant and imatrix quantization levels, RAM/VRAM sharing and managing models visually through LM Studio.

Participants launch a local OpenAI-compatible API with llama-server and LM Studio on Windows, macOS (Apple Silicon) or Linux workstations, tune GPU offloading for maximum token speed and run confidential document analysis workflows locally.

Who Should Attend?

  • Software developers who want to run local LLMs on their own workstations
  • Data analysts and researchers working with sensitive documents off the cloud
  • Technical leads defining a standard local AI setup for developer machines
  • Prerequisite: basic computer and command-line familiarity

Key Takeaways

  • Select GGUF formats and quantization levels such as Q4_K_M, Q5_K_M, Q8_0 and IQ4_XS
  • Configure CPU, CUDA, Apple Metal and Vulkan acceleration with llama.cpp
  • Balance RAM and VRAM using GPU layer offloading and context length parameters
  • Expose a local OpenAI-compatible endpoint via LM Studio and llama-server
  • Produce structured output on the desktop using grammar and JSON schema constraints

Curriculum

The GGUF Format and Quantization Levels
  • GGUF metadata structure, tokenizer and single-file model distribution
  • K-quants (Q4_K_M, Q5_K_M, Q6_K, Q8_0) and imatrix quality/memory trade-offs
llama.cpp Setup and Hardware Acceleration
  • CUDA, Metal, Vulkan and AVX2/AMX build and runtime options
  • -ngl (GPU layers), -c (context), flash attention and KV cache quantization
Local API and Structured Output with llama-server
  • OpenAI-compatible endpoint, parallel slots and speculative decoding
  • Deterministic output with GBNF grammars and JSON schemas
Desktop Workflows with LM Studio
  • Model discovery, hardware fit checks and preset management
  • Local API server, document chat and MCP/application integration
Benchmarking and Workstation Sizing Guide
  • Measuring prompt processing and token generation with llama-bench
  • Hardware and model recommendation matrix for developer profiles

Other Courses in This Category

  • Self-Hosted AI Coding Assistants with Local LLMs

    Building self-hosted AI coding assistants without sending source code outside the company: Qwen2.5-Coder and DeepSeek-Coder models, FIM autocomplete, Continue.dev/Tabby IDE plugins, vLLM serving and codebase RAG integration.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Open-Weight LLM Selection, Licensing and Benchmarking for Enterprise Use

    Selecting open-weight LLMs for enterprise use: Llama, Qwen, Mistral, Gemma and DeepSeek families; commercial licence analysis, dense vs MoE trade-offs, Turkish and tool-calling capability, and building a benchmark matrix on company data.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Enterprise Model Serving with NVIDIA Triton Inference Server and KServe

    Multi-model GPU serving with NVIDIA Triton Inference Server and KServe: TensorRT-LLM, ONNX and Python backends, dynamic batching, model ensembles, KServe InferenceService, Knative autoscaling and tuning with Model Analyzer.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • KV Cache Management and Continuous Batching Optimization for LLM Inference

    Solving memory and concurrency bottlenecks in LLM serving: KV cache memory math, PagedAttention, SGLang RadixAttention, prefix caching, FP8 KV cache quantization, continuous batching and chunked prefill tuning.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • LLM Inference Throughput and Latency Optimization

    Optimising TTFT, ITL and total throughput in LLM services: compute-bound vs memory-bandwidth-bound bottleneck analysis, FlashAttention/FlashInfer, CUDA Graphs, speculative decoding, parallelism tuning and SLO-driven load testing.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Running AI and LLM Workloads on Kubernetes

    Operating GPU and LLM workloads on Kubernetes: NVIDIA GPU Operator, GPU sharing with MIG and time-slicing, queue management with Kueue/Volcano, fast model weight caching, GPU autoscaling and DCGM observability.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View

Get in touch