Course Objective & Scope
Letting individual teams wire applications directly to OpenAI, Anthropic, Gemini, Azure/Bedrock or internal vLLM endpoints leads to scattered API keys, brittle outages when a provider throttles traffic and unmonitored spend. This course teaches how to place a central Model Gateway (LiteLLM Proxy, Envoy AI Gateway, Kong/Portkey) between applications and model providers and design intelligent routing policies.
In hands-on labs participants unify cloud and local models behind a single OpenAI-compatible API, configure automated fallbacks on timeouts and HTTP 429s, set up latency-aware load balancing and complexity-based routing, and enforce per-team quotas and budgets.
Who Should Attend?
- Platform engineers building shared internal AI infrastructure and service layers
- Software and solution architects working with multiple model providers
- MLOps and SRE teams responsible for API security, quotas and cost governance
- Prerequisite: Docker, REST API management and basic LLM service concepts
Key Takeaways
- Expose cloud APIs and in-house vLLM/Ollama services through one standard schema
- Build seamless cross-provider fallback chains for rate limits (429) and outages
- Route requests to the right model based on cost, latency, data sensitivity or task complexity
- Define virtual API keys, RBAC and department-level spend caps
- Operate central logging, PII redaction and caching policies at the gateway layer
Curriculum
Model Gateway Architecture and Tooling Landscape
- How AI gateways differ from classic API gateways (token-aware rate limits, SSE streaming)
- Comparing LiteLLM Proxy, Envoy AI Gateway, Kong AI Gateway and Portkey
Unified API, Load Balancing and Resilience
- Unifying OpenAI, Anthropic, Bedrock, Vertex and internal vLLM models behind one endpoint
- Retries, cooldowns, circuit breakers and cross-region/cross-provider fallbacks
Intelligent and Semantic Routing
- Latency-based, least-busy and cost-optimised routing strategies
- Routing by query complexity and data sensitivity using lightweight classifiers
Authentication, Virtual Keys and Quota Governance
- Enterprise SSO/OIDC integration, team virtual keys and model allowlists
- Tokens/requests per minute (TPM/RPM) limits and monthly hard/soft budget caps
Centralised Observability and High Availability
- Streaming OpenTelemetry/Langfuse traces and cost attribution directly from the gateway
- High-availability multi-replica gateway deployment with PostgreSQL and Redis
Other Courses in This Category
-
GitHub Actions and GitLab CI pipelines for LLM and RAG applications: fast PR smoke and eval suites, deterministic and model-graded quality gates, cost and latency budget checks, canary rollouts and automated rollbacks.
- Live Course
- 5 modules
-
Systematic quality measurement for LLM applications: deterministic and LLM-as-a-judge metrics with Promptfoo, DeepEval and Ragas, mitigating judge biases, calibrating against human labels and building an enterprise eval scorecard.
- Live Course
- 5 modules
-
Engineering fast, repeatable, large-scale LLM evaluation pipelines: async parallel test runners, judge cost and cache optimisation, dataset ingestion, statistical significance and experiment comparison dashboards.
- Live Course
- 5 modules
-
Continuous quality monitoring on live LLM traffic: smart trace sampling, zero-latency async online evaluators, reference-free hallucination and relevance scoring, query drift detection and automated alerting.
- Live Course
- 5 modules
-
Preventing silent breakages when prompts, models or RAG settings change: deterministic contract assertions, semantic equivalence checks, tool-call verification, snapshot diffs and release sign-off matrices.
- Live Course
- 5 modules
-
Deep observability for multi-step LLM, RAG and agent systems: OpenTelemetry GenAI conventions, self-hosted Langfuse and Arize Phoenix, per-span token/latency/cost breakdown, PII masking and production trace dashboards.
- Live Course
- 5 modules
Get in touch
