Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Factory floors, in-vehicle systems, retail terminals, mobile devices and branch locations with intermittent connectivity cannot rely on cloud or central data-centre LLMs. This course covers how to select, compress and deploy 1B–4B Small Language Models (SLMs) on edge hardware under strict memory, power and thermal limits.

Participants compile compact Llama, Qwen, Gemma and Phi models using ONNX Runtime GenAI, ExecuTorch, MLC-LLM and llama.cpp, and design an edge deployment blueprint that delivers reliable outputs for focused tasks such as command parsing, form extraction and offline voice/text assistance.

Who Should Attend?

  • Software engineers building IoT, embedded, mobile and field applications
  • ML engineers working on edge AI hardware (NVIDIA Jetson, industrial PCs, NPU-equipped devices)
  • Solution architects designing offline assistants for branches, plants or field operations
  • Prerequisite: Python or C++/mobile development basics and foundational model inference knowledge

Key Takeaways

  • Select 1B–4B Small Language Models (SLMs) matched to edge hardware constraints
  • Compile models for target hardware using ONNX Runtime GenAI, ExecuTorch and llama.cpp
  • Fit tight RAM budgets with 4-bit quantization and bounded context windows
  • Curb SLM hallucinations using grammar constraints and task-focused prompts
  • Design secure over-the-air model distribution and hybrid edge-plus-cloud fallback architectures

Curriculum

Edge LLM Use Cases and SLM Selection
  • Why edge: zero network latency, offline continuity and data privacy
  • Task performance of Llama 3.2 (1B/3B), Qwen2.5 (0.5B–3B), Gemma and Phi models
Target Hardware and the Edge Runtime Ecosystem
  • NVIDIA Jetson (Orin), Apple Neural Engine, Qualcomm/Intel NPUs and industrial PCs
  • Comparing ONNX Runtime GenAI, ExecuTorch, MLC-LLM, TensorRT and llama.cpp
Optimising Under Memory, Power and Thermal Limits
  • INT4/AWQ/GGUF compression, unified memory budgets and KV cache caps
  • Preventing thermal throttling, batch-size-1 latency tuning and power measurement
Engineering Reliability in Small Models
  • Single-purpose prompts, JSON/grammar enforcement and local function calling
  • Lightweight on-device RAG with an SLM and local SQLite/vector search
Fleet Rollout and Hybrid Edge-Cloud Architecture
  • Delta model updates and version control across field devices
  • Hybrid routing that escalates low-confidence requests to a central server

Other Courses in This Category

  • LLM Unit Economics: Cost per Request and Token Optimization

    Unit cost analysis and optimisation in LLM systems: self-hosted GPU vs commercial API break-even math, prompt and output token reduction, batch APIs, model cascading and AI FinOps metrics.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • LLM Model Serving and Deployment Fundamentals

    Shows how an open-weight LLM is served as an OpenAI-compatible API on a GPU server: inference engines, VRAM math, core serving settings and a first benchmark. An entry-level programme for developer and platform teams.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • LLM Serving with vLLM: Production Inference Deployment

    High-throughput LLM serving with vLLM: PagedAttention, continuous batching, tensor parallelism, prefix caching, multi-LoRA serving and Prometheus metrics. Teams leave with a production-ready vLLM deployment and benchmark report.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Hugging Face Text Generation Inference (TGI) for LLM Serving

    Serving LLMs with Hugging Face Text Generation Inference (TGI): Docker setup, sharding, quantization, Messages API, structured output with guidance and Inference Endpoints. A hands-on programme for teams in the Hugging Face ecosystem.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Running Local LLMs with Ollama

    Running open-weight models on laptops, workstations or servers with Ollama: model library, Modelfile, REST/OpenAI-compatible API, Open WebUI, embeddings and a shared team setup. For fast prototyping and confidential-data scenarios.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • LLM Serving Stack Mastery: FastAPI, vLLM, KServe and Triton

    Combines a FastAPI model API layer, vLLM for LLM inference, KServe for model management on Kubernetes and Triton for multi-framework serving in one architecture. Participants build an end-to-end, scalable model serving stack.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    6 modules
    View

Get in touch