Model Online / On-Site
Duration
Modules 6 modules

Course Objective & Scope

Saying that a RAG system's "quality dropped" is not actionable unless you can isolate whether the retriever missed the relevant passage (Recall@k), ranked it too low (MRR/NDCG), or the generator failed to synthesise the evidence faithfully (Faithfulness/Answer Relevance). This course provides a rigorous look at classical Information Retrieval (IR) metrics alongside modern LLM-based generation and citation metrics.

Participants compute reference-based and reference-free metrics in Python, analyse metric correlations and Failure-Mode matrices, and define quantitative release thresholds for their organisation's RAG services.

Who Should Attend?

  • Data scientists and AI engineers optimising search and RAG quality through quantitative metrics
  • ML and search engineers running A/B and offline benchmarks across embeddings and rerankers
  • Technical leads defining quality KPIs and acceptance criteria for AI products
  • Prerequisite: basic Python, familiarity with statistics and working knowledge of RAG

Key Takeaways

  • Select and compute Recall@k, Precision@k, Hit Rate, MRR, MAP and NDCG@k for retrieval tasks
  • Implement the RAG Triad metrics (Context Relevance, Faithfulness/Groundedness, Answer Relevance)
  • Distinguish when to use ground-truth reference metrics versus reference-free production metrics
  • Measure attribution quality at the claim level using Citation Precision, Citation Recall and F1
  • Use bootstrap confidence intervals and paired significance tests when comparing two RAG pipelines

Curriculum

Unranked Retrieval Metrics: Recall@k, Precision@k and Hit Rate
  • Document-level, chunk-level and character-span matching definitions
  • Computing full-evidence coverage vs partial recall on multi-hop questions
Rank-Aware IR Metrics: MRR, MAP and NDCG@k
  • MRR for single-target lookup, MAP for multi-target sets and NDCG@k for graded relevance (0–3)
  • Reading `Recall@50` alongside `NDCG@5` to separate retriever recall from reranker precision
The RAG Triad and Context Quality Metrics
  • Context Precision, Context Recall and context signal-to-noise ratios
  • Claim-decomposition-based Faithfulness and Answer Relevance scoring
Reference-Based Accuracy and Citation Metrics
  • Factual Correctness against reference answers, semantic similarity and completeness
  • Citation Precision (penalising irrelevant citations) and Citation Recall (flagging unsupported claims)
Statistical Significance, Calibration and Online Proxy Metrics
  • Accounting for judge variance with bootstrap confidence intervals and paired significance tests
  • Correlating reference-free online metrics with implicit user signals (reformulations, copy actions, feedback)
Lab: Building a RAG Diagnostic Scorecard
  • Diagnostic decision matrix combining retrieval and generation metrics
  • Defining release quality thresholds (SLOs) for an internal RAG service

Other Courses in This Category

  • Production RAG Architecture: Scaling, Freshness, Security and Operations

    Scaling RAG from prototype to production: event-driven incremental ingestion, zero-downtime re-indexing, multi-tier caching, security controls, latency SLAs and LLMOps observability.

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View
  • Building an Enterprise Knowledge Assistant: Architecture and Implementation

    End-to-end engineering programme for building an enterprise knowledge assistant across SharePoint, Confluence, Jira and internal docs with multi-turn memory, citations, permissions and feedback loops.

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View
  • Text-to-SQL and Conversational Analytics over Enterprise Databases

    Engineering Text-to-SQL and conversational analytics systems that translate natural-language business questions into accurate SQL queries, tables and charts over PostgreSQL, BigQuery, Snowflake and SQL Server.

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View
  • Semantic Layers and Governed Metric Definitions for Text-to-SQL and AI Analytics

    Preventing inconsistent numbers in AI analytics by grounding LLMs in a governed semantic layer (Cube, dbt Semantic Layer, YAML metric definitions, canonical joins and business glossaries).

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View
  • SQL Query Validation, Guardrails and Access Control for Text-to-SQL

    Securing production Text-to-SQL systems with AST validation (SQLGlot), read-only execution sandboxes, Row-Level Security (RLS), column masking, query cost guards and semantic verification.

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View
  • Text-to-SQL Accuracy Evaluation and Enterprise Benchmarking

    Evaluating Text-to-SQL systems rigorously using Execution Accuracy, result-set matching, AST structural comparison, query efficiency scoring and enterprise benchmark construction.

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View

Get in touch