Course Objective & Scope
In enterprise assistants and customer support bots, a large fraction of users ask identical or near-identical questions, while every request repeatedly processes thousands of tokens of identical system prompts, tool definitions and document context. This course teaches how to layer exact caching, semantic caching, prefix/prompt caching and embedding caching to slash both compute spend and response latency.
In hands-on labs participants build a semantic cache with Redis/Qdrant, restructure prompt templates to maximise prompt-cache hit rates on cloud APIs and vLLM, and calibrate similarity thresholds to prevent false-positive cache hits.
Who Should Attend?
- Software and AI engineers building high-traffic LLM and RAG services
- Platform and MLOps engineers managing model gateways and caching infrastructure
- Solution architects looking to reduce API spend and response latency
- Prerequisite: Python, basic Redis or vector database familiarity and LLM API experience
Key Takeaways
- Structure prompts with static prefixes first and dynamic inputs last to maximise prompt-cache hits
- Implement deterministic exact caching for LLM completions and embeddings using Redis
- Design a low-risk semantic cache using vector similarity plus cross-encoder verification
- Enforce tenant/role namespaces and TTL invalidation rules to prevent cross-user data leaks
- Measure cache hit rate, false-positive rate and saved tokens and latency
Curriculum
LLM Caching Layers: What to Cache Where
- Exact response cache, semantic cache, KV/prompt prefix cache and embedding/retrieval cache
- Matching workloads (FAQ bots, RAG, code completion, personalised agents) to the right cache type
Prompt and Prefix Caching Engineering
- How prefix caching works in OpenAI, Anthropic, Gemini and vLLM APC
- Ordering system prompts, tool schemas, few-shot examples and dynamic timestamps/user context
Exact Caching and Embedding/Retrieval Caches
- Normalized request hashing for exact-match caching with LiteLLM and Redis
- Caching embedding vectors and reranker scores for repeated queries and documents
Semantic Cache Design and Preventing False Positives
- Why naive semantic caches fail on numbers, dates, negations and near-homonyms
- Combining strict similarity thresholds, entity matching and lightweight rerankers
Security, Cache Invalidation and Savings Measurement
- Scoping cache keys by tenant and ACL permissions to prevent cross-user data leakage
- Invalidating cached answers when source documents change and tracking net cost savings
Other Courses in This Category
-
End-to-end latency management in multi-step LLM, RAG and agent applications: SSE streaming UX, parallel retrieval/guardrail/tool execution, output token reduction, P95/P99 tail-latency control and hedged request patterns.
- Live Course
- 5 modules
-
Incident management tailored to production LLM and agent systems: response runbooks for hallucination spikes, prompt injection/data leaks, runaway agent loops and provider outages, kill-switch controls and AI post-mortems.
- Live Course
- 5 modules
-
Combining managed cloud AI platforms (Azure AI Foundry, Amazon Bedrock, Google Vertex AI) with on-premises GPU clusters in a hybrid architecture: sensitivity-based routing, portable contracts, private networking and disaster recovery.
- Live Course
- 5 modules
-
Moving from ad-hoc 'vibe checks' to measurable quality in LLM applications: building a first eval dataset, rule-based assertions, basic LLM-as-a-judge grading and running a first automated evaluation report with Promptfoo.
- Live Course
- 5 modules
-
Designing high-quality golden datasets that anchor LLM evaluations: scenario matrices, synthetic test generation and filtering from documents, adversarial/edge-case design, expert annotation protocols and dataset maintenance.
- Live Course
- 5 modules
-
Hybrid evaluator design combining deterministic rule-based assertions and model-graded (LLM-as-a-judge) scoring: regex, AST and JSON schema checks, pairwise and rubric-based judge prompts, bias control and calibration.
- Live Course
- 5 modules
Get in touch
