Model Online / On-Site
Duration
Modules 6 modules

Course Objective & Scope

Saying that a RAG system's "quality dropped" is not actionable unless you can isolate whether the retriever missed the relevant passage (Recall@k), ranked it too low (MRR/NDCG), or the generator failed to synthesise the evidence faithfully (Faithfulness/Answer Relevance). This course provides a rigorous look at classical Information Retrieval (IR) metrics alongside modern LLM-based generation and citation metrics.

Participants compute reference-based and reference-free metrics in Python, analyse metric correlations and Failure-Mode matrices, and define quantitative release thresholds for their organisation's RAG services.

Who Should Attend?

  • Data scientists and AI engineers optimising search and RAG quality through quantitative metrics
  • ML and search engineers running A/B and offline benchmarks across embeddings and rerankers
  • Technical leads defining quality KPIs and acceptance criteria for AI products
  • Prerequisite: basic Python, familiarity with statistics and working knowledge of RAG

Key Takeaways

  • Select and compute Recall@k, Precision@k, Hit Rate, MRR, MAP and NDCG@k for retrieval tasks
  • Implement the RAG Triad metrics (Context Relevance, Faithfulness/Groundedness, Answer Relevance)
  • Distinguish when to use ground-truth reference metrics versus reference-free production metrics
  • Measure attribution quality at the claim level using Citation Precision, Citation Recall and F1
  • Use bootstrap confidence intervals and paired significance tests when comparing two RAG pipelines

Curriculum

Unranked Retrieval Metrics: Recall@k, Precision@k and Hit Rate
  • Document-level, chunk-level and character-span matching definitions
  • Computing full-evidence coverage vs partial recall on multi-hop questions
Rank-Aware IR Metrics: MRR, MAP and NDCG@k
  • MRR for single-target lookup, MAP for multi-target sets and NDCG@k for graded relevance (0–3)
  • Reading `Recall@50` alongside `NDCG@5` to separate retriever recall from reranker precision
The RAG Triad and Context Quality Metrics
  • Context Precision, Context Recall and context signal-to-noise ratios
  • Claim-decomposition-based Faithfulness and Answer Relevance scoring
Reference-Based Accuracy and Citation Metrics
  • Factual Correctness against reference answers, semantic similarity and completeness
  • Citation Precision (penalising irrelevant citations) and Citation Recall (flagging unsupported claims)
Statistical Significance, Calibration and Online Proxy Metrics
  • Accounting for judge variance with bootstrap confidence intervals and paired significance tests
  • Correlating reference-free online metrics with implicit user signals (reformulations, copy actions, feedback)
Lab: Building a RAG Diagnostic Scorecard
  • Diagnostic decision matrix combining retrieval and generation metrics
  • Defining release quality thresholds (SLOs) for an internal RAG service

Other Courses in This Category

Get in touch