Course Objective & Scope
Pre-release static test sets cannot anticipate shifting user behaviour, seasonal product launches or malformed documents ingested into a live knowledge base. Because production traffic has no ground-truth reference answers, evaluating live systems requires a fundamentally different architecture. This advanced programme teaches how to run continuous evaluation on live traffic without adding latency to user requests.
Participants build an asynchronous pipeline that consumes production traces from a queue, applies stratified and signal-based sampling, runs reference-free RAG and agent evaluators, and triggers alerts when answer quality or query topics drift.
Who Should Attend?
- LLMOps, MLOps and SRE engineers operating high-traffic LLM applications in production
- Data scientists and AI engineers monitoring live quality and model drift
- Infrastructure teams building observability and quality platforms
- Prerequisite: LLM evaluation fundamentals, tracing concepts and Python proficiency
Key Takeaways
- Architect asynchronous online evaluation workers that add zero latency to user responses
- Control evaluation spend using stratified, rule-based and signal-triggered trace sampling
- Score faithfulness, toxicity and task completion on live traffic without ground-truth labels
- Detect emerging topics and query drift using embedding clustering and distribution metrics
- Route low-scoring production traces automatically into human annotation queues
Curriculum
Offline Evaluation versus Continuous Online Evaluation
- Operating without ground truth, managing judge budgets and isolating user latency
- Out-of-band trace ingestion and message-queue worker architecture
Smart Trace Sampling Strategies
- Moving beyond uniform random sampling to target high-latency, multi-turn or low-retrieval-score traces
- Triggering 100% evaluation on negative user signals (thumbs-down, rephrasing, abandonment)
Reference-Free Production Evaluators
- Checking RAG faithfulness (hallucination against retrieved context) and answer relevance
- Evaluating policy compliance, PII leakage, brand tone and tool-call errors
Query Drift and Knowledge Gap Detection
- Monitoring incoming query embedding distributions and emerging topic clusters
- Identifying unanswered user intents where the knowledge base lacks coverage
Quality Alerting and Human Annotation Queues
- Moving-average and anomaly alerts for quality SLO burn rates
- Routing low-scoring traces into Langfuse/Phoenix annotation queues and regression suites
Other Courses in This Category
-
Preventing silent breakages when prompts, models or RAG settings change: deterministic contract assertions, semantic equivalence checks, tool-call verification, snapshot diffs and release sign-off matrices.
- Live Course
- 5 modules
-
Deep observability for multi-step LLM, RAG and agent systems: OpenTelemetry GenAI conventions, self-hosted Langfuse and Arize Phoenix, per-span token/latency/cost breakdown, PII masking and production trace dashboards.
- Live Course
- 5 modules
-
Low-latency input and output guardrails for production LLM applications: blocking prompt injection and jailbreaks, KVKK/GDPR-compliant PII masking, topic rails, hallucination checks and safe tool execution controls.
- Live Course
- 5 modules
-
Cutting cost and latency in production LLM applications with multi-layer caching: provider and vLLM prompt caching, Redis exact caching, vector-based semantic caching, tenant isolation and cache invalidation.
- Live Course
- 5 modules
-
End-to-end latency management in multi-step LLM, RAG and agent applications: SSE streaming UX, parallel retrieval/guardrail/tool execution, output token reduction, P95/P99 tail-latency control and hedged request patterns.
- Live Course
- 5 modules
-
Incident management tailored to production LLM and agent systems: response runbooks for hallucination spikes, prompt injection/data leaks, runaway agent loops and provider outages, kill-switch controls and AI post-mortems.
- Live Course
- 5 modules
Get in touch
