Course Objective & Scope
When a RAG pipeline or autonomous agent produces a bad answer, inspecting only the final response hides whether the failure came from missing the right document during retrieval, misreading retrieved context during generation or calling the wrong tool with bad arguments. This course teaches how to evaluate multi-component RAG and agent systems layer by layer.
Using Ragas, DeepEval and Langfuse/Phoenix, participants measure retrieval, reranking, grounding, tool trajectories and task completion independently to pinpoint the exact component holding back system accuracy.
Who Should Attend?
- AI and software engineers building enterprise RAG and multi-step agent systems
- Data scientists working on search/retrieval quality and agent reliability
- LLMOps and quality engineers validating complex GenAI workflows
- Prerequisite: hands-on experience building RAG and tool-calling/agent applications in Python
Key Takeaways
- Isolate RAG failures cleanly between the retrieval stage and the generation stage
- Evaluate chunking, embedding and reranker changes with Recall@k, Precision@k and NDCG
- Measure context faithfulness (groundedness) and citation accuracy in generated answers
- Score agent tool selection, JSON argument correctness and trajectory step efficiency
- Test multi-turn agents end to end using simulated users and deterministic mock tools
Curriculum
The RAG Evaluation Triad and Failure Isolation
- Relationships between the user query, retrieved context and generated answer
- Diagnosing whether retrieval missed the facts or generation ignored them
Retrieval and Reranking Metrics
- Measuring search and reranker quality with Hit Rate, Recall@k, Precision@k, MRR and NDCG
- Benchmarking query rewriting, hybrid search (BM25 + dense) and chunking experiments
Generation, Grounding and Citation Evaluation
- Claim-level faithfulness (hallucination detection) and context utilisation scoring
- Verifying that inline citations (
[1],[2]) genuinely support the stated claims
Single-Step and Multi-Step Agent Evaluation
- Tool selection accuracy and JSON argument validation
- Trajectory evaluation: optimal path matching, redundant tool calls and loop detection
Sandboxed Environments, Mock Tools and User Simulation
- Deterministic mock APIs and final database/state verification without production side effects
- Persona-driven user simulators for automated multi-turn agent testing
Other Courses in This Category
-
Connecting pre-release offline eval scores with live production outcomes: offline-to-online correlation, shadow deployments, A/B testing and interleaving for LLM responses, business metric alignment and staged rollouts.
- Live Course
- 5 modules
-
Turning user interactions into continuous improvement in production LLM products: explicit and implicit feedback capture, trace-linked scoring, expert correction queues and building an eval and fine-tuning data flywheel.
- Live Course
- 5 modules
-
Fundamentals of moving prototype LLM applications into production: how LLMOps differs from classic MLOps, prompt and model versioning, basic tracing, starter eval datasets, guardrails and a production readiness checklist.
- Live Course
- 5 modules
-
Bringing full production discipline to an LLM application: LiteLLM gateway, Langfuse/Phoenix tracing, automated eval and regression tests, CI/CD quality gates, runtime guardrails and live user feedback loops.
- Live Course
- 5 modules
-
Full reproducibility in LLM applications: prompt versioning with Langfuse/MLflow Prompt Registry and Git, model/LoRA and tool schema versioning, RAG/eval dataset snapshots and safe rollbacks.
- Live Course
- 5 modules
-
Consolidating cloud and self-hosted LLMs behind a single OpenAI-compatible gateway with LiteLLM and enterprise AI gateways: automated fallbacks, load balancing, complexity-based routing, virtual keys and budget/rate-limit governance.
- Live Course
- 5 modules
Get in touch
