Model Online / On-Site
Duration
Modules 6 modules

Course Objective & Scope

An LLM workflow that works in a demo can easily fail in production due to variable latency, provider outages, rate-limit spikes, non-deterministic outputs and security risks. This course covers reference architecture patterns for high-availability LLM systems: centralised AI gateways, synchronous vs event-driven processing, circuit breakers and multi-tier fallback strategies.

Participants design an end-to-end production architecture that integrates cleanly with their organisation's existing microservices and data platform, complete with failure/fallback design and OpenTelemetry observability.

Who Should Attend?

  • Software, platform and solution architects responsible for production LLM architectures
  • Senior backend and AI engineers integrating AI services into mission-critical workflows
  • Engineering leaders establishing internal AI platforms, standards and SRE practices
  • Prerequisite: experience with distributed systems/microservices and LLM API development

Key Takeaways

  • Select between synchronous, streaming and event-driven queue architectures per workload
  • Design a central AI gateway handling authentication, quotas, PII redaction and routing
  • Apply timeouts, retry budgets, circuit breakers and graceful degradation patterns
  • Embed input and output guardrails without blowing the latency budget
  • Monitor production health using OpenTelemetry distributed tracing and online eval sampling

Curriculum

From Prototype to Production: Distributed Systems Challenges of LLMs
  • Architectural impact of high tail latency, token-based quotas and stochastic failures
  • Synchronous APIs, SSE streaming and Kafka/RabbitMQ event-driven workflow patterns
Centralised AI Gateway and Control Plane
  • Single entry point across teams and providers: auth, quotas and cost chargeback
  • Central prompt/model registries and shadow or canary traffic routing
Resilience, Failure and Fallback Design
  • Request hedging, tiered timeouts, avoiding retry storms and circuit breakers
  • Cross-region/cross-provider fallbacks and deterministic non-LLM degradation paths
State Management, Idempotency and Caching
  • State persistence, checkpointing and idempotent tool execution in multi-step flows
  • Multi-tier caching architectures (prompt cache, exact cache and semantic cache)
Security, Guardrail and Observability Layers
  • Decoupling fast inline guardrails from asynchronous deep inspection
  • OpenTelemetry GenAI semantic conventions, Langfuse/Arize Phoenix and online eval sampling
Architecture Workshop: Enterprise Production Reference Design
  • Producing an Architecture Decision Record (ADR) and component blueprint for an internal product
  • Design review against real-world failure modes and outage scenarios

Other Courses in This Category

  • Large Language Model (LLM) Fundamentals

    How foundation models and LLMs work: tokens, context windows, sampling, hallucination, model families and when to choose prompting, RAG or fine-tuning — a hands-on starting point for technical teams.

    LLM Application Engineering Corporate
    Live Course
    6 modules
    View
  • Large Language Models and the Transformer Architecture

    A technical deep dive into the transformer: tokenization, self-attention, RoPE, KV cache, scaling laws, SFT, RLHF/DPO and reasoning models, explored hands-on with PyTorch and Hugging Face Transformers.

    LLM Application Engineering Corporate
    Live Course
    6 modules
    View
  • Building Applications with LLM APIs

    Hands-on programme for building a near-production LLM application with the OpenAI, Anthropic Claude and Google Gemini APIs, covering streaming, conversation state, retries, rate limits, fallbacks and basic evals.

    LLM Application Engineering Corporate
    Live Course
    6 modules
    View
  • Structured Outputs with LLMs: JSON Schema, Pydantic and Constrained Decoding

    Getting reliable JSON from LLMs with JSON Schema, OpenAI Structured Outputs, Gemini response schemas, Pydantic/Zod, Instructor and constrained decoding, plus validation, repair loops and document data extraction.

    LLM Application Engineering Corporate
    Live Course
    6 modules
    View
  • LLM Tool Use and Function Calling

    How to let LLMs safely call APIs, databases and internal systems via tool/function calling: tool schema design, parallel calls, MCP servers, error handling and permission boundaries.

    LLM Application Engineering Corporate
    Live Course
    6 modules
    View
  • Prompt and Context Architecture for LLM Applications

    Context engineering for LLM applications: designing how system prompts, templates, few-shot examples, retrieved content, memory and tool definitions fill the context window, with versioning and eval-driven iteration.

    LLM Application Engineering Corporate
    Live Course
    6 modules
    View

Get in touch