Model Online / On-Site
Duration
Modules 6 modules

Course Objective & Scope

Many RAG projects stall before production because teams rely on ad-hoc "vibe checks" over a handful of prompts, making it impossible to know whether a change to chunking, embeddings or prompts helped or caused regressions. This course covers building a complete RAG evaluation lifecycle: curating representative golden datasets, generating synthetic edge cases, running automated suites with RAGAS, DeepEval and Promptfoo, and performing root-cause error analysis.

Participants build a golden evaluation dataset from their own corporate documents, wire an automated eval suite into CI/CD, and design a production feedback flywheel.

Who Should Attend?

  • AI and software engineers taking RAG and enterprise knowledge assistants into production
  • Data scientists running systematic experiments across chunking, retrieval and prompting
  • QA, MLOps and technical leads establishing quality gates and regression suites for AI applications
  • Prerequisite: basic RAG architecture knowledge and Python development experience

Key Takeaways

  • Curate a balanced golden dataset combining real queries, domain-expert annotations and synthetic generation
  • Cover single-fact, multi-hop, ambiguous, conflicting and unanswerable question categories
  • Evaluate retrieval and generation components independently using RAGAS, DeepEval or Promptfoo
  • Align LLM-as-a-judge evaluators with human judgements and mitigate judge biases
  • Set up CI/CD regression gates and a production trace feedback loop that continuously grows the test set

Curriculum

Beyond Vibe Checks: The RAG Evaluation Lifecycle
  • Isolating failure sources across data ingestion, retrieval and answer generation
  • Connecting offline experimentation, CI/CD regression suites and online production monitoring
Golden Dataset Curation and Synthetic Test Generation
  • Generating controlled synthetic (question, ground-truth chunk, reference answer) triples
  • Stratifying test cases across single-fact, tabular, multi-hop and unanswerable queries
LLM-as-a-Judge Design and Human Alignment
  • Binary pass/fail and rubric-based judge prompts; mitigating verbosity and position bias
  • Calibrating automated judges against a small set of Subject Matter Expert (SME) labels
Evaluation Tooling: RAGAS, DeepEval, Promptfoo and Tracing Platforms
  • Running pytest-style suites with DeepEval and RAGAS, and declarative evaluations with Promptfoo
  • Tracking experiment versions and comparing retrieval/prompt configurations in Langfuse or Phoenix
Failure Taxonomy, Root-Cause Analysis and the Data Flywheel
  • Bucketing failures into chunking splits, filter errors, ranking misses or synthesis hallucinations
  • Converting production thumbs-down and low-confidence traces into regression test cases
Hands-On: End-to-End Evaluation Pipeline for Enterprise RAG
  • Building a stratified evaluation dataset and automated test runner on internal documents
  • Producing an evidence-backed comparison report between two RAG configurations

Other Courses in This Category

  • Retrieval and Generation Metrics for RAG Systems

    Hands-on programme covering information retrieval metrics (Recall@k, Precision@k, MRR, NDCG@k) and generation metrics (Faithfulness, Answer Relevance, Context Precision, Citation F1) with statistical testing.

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View
  • Production RAG Architecture: Scaling, Freshness, Security and Operations

    Scaling RAG from prototype to production: event-driven incremental ingestion, zero-downtime re-indexing, multi-tier caching, security controls, latency SLAs and LLMOps observability.

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View
  • Building an Enterprise Knowledge Assistant: Architecture and Implementation

    End-to-end engineering programme for building an enterprise knowledge assistant across SharePoint, Confluence, Jira and internal docs with multi-turn memory, citations, permissions and feedback loops.

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View
  • Text-to-SQL and Conversational Analytics over Enterprise Databases

    Engineering Text-to-SQL and conversational analytics systems that translate natural-language business questions into accurate SQL queries, tables and charts over PostgreSQL, BigQuery, Snowflake and SQL Server.

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View
  • Semantic Layers and Governed Metric Definitions for Text-to-SQL and AI Analytics

    Preventing inconsistent numbers in AI analytics by grounding LLMs in a governed semantic layer (Cube, dbt Semantic Layer, YAML metric definitions, canonical joins and business glossaries).

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View
  • SQL Query Validation, Guardrails and Access Control for Text-to-SQL

    Securing production Text-to-SQL systems with AST validation (SQLGlot), read-only execution sandboxes, Row-Level Security (RLS), column masking, query cost guards and semantic verification.

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View

Get in touch