Course Objective & Scope
Enterprise AI systems cannot be governed by 'we tried a few prompts and the answers looked fine', yet legacy text metrics like BLEU and ROUGE fail to capture semantic quality in modern LLM tasks. This programme teaches how to choose the right evaluation metrics per task type, design reliable LLM-as-a-judge rubrics and align automated judges with human domain experts.
Using Promptfoo, DeepEval and Ragas, participants implement reference-based and reference-free evaluators for their own use cases, test for verbosity and position bias, and produce an evaluation scorecard suitable for release decisions.
Who Should Attend?
- AI engineers and data scientists accountable for LLM, RAG and agent quality
- Software developers who need objective metrics to validate model and prompt changes
- Technical product managers and QA engineers defining acceptance criteria
- Prerequisite: Python and basic experience with LLM APIs
Key Takeaways
- Select appropriate metrics for summarisation, classification, extraction and Q&A tasks
- Write binary (pass/fail) and low-granularity LLM-as-a-judge rubrics aligned with domain experts
- Mitigate position, verbosity and self-preference biases in judge models
- Measure agreement between human annotations and automated judges (Cohen's kappa / correlation)
- Build a repeatable enterprise evaluation suite with Promptfoo or DeepEval
Curriculum
Taxonomy of LLM Evaluation
- Reference-based (ground-truth) versus reference-free evaluation approaches
- Code-based deterministic checks, embedding similarity and LLM-as-a-judge
Error Analysis and Task-Specific Metric Design
- Categorising real application failure modes with open and axial coding
- Replacing vague 'helpfulness' scores with concrete, domain-specific criteria
LLM-as-a-Judge Engineering and Bias Mitigation
- Pointwise and pairwise grading, G-Eval and critique-first scoring
- Detecting and reducing position, verbosity and self-preference biases
Human-in-the-Loop Calibration
- Expert annotation workflows and measuring inter-annotator agreement
- Calibrating judge prompts against human labels to track TPR and TNR
Hands-On Implementation with Evaluation Frameworks
- Comparing Promptfoo, DeepEval and Ragas architectures
- Running multi-model/multi-prompt experiment matrices and generating a release scorecard
Other Courses in This Category
-
Engineering fast, repeatable, large-scale LLM evaluation pipelines: async parallel test runners, judge cost and cache optimisation, dataset ingestion, statistical significance and experiment comparison dashboards.
- Live Course
- 5 modules
-
Continuous quality monitoring on live LLM traffic: smart trace sampling, zero-latency async online evaluators, reference-free hallucination and relevance scoring, query drift detection and automated alerting.
- Live Course
- 5 modules
-
Preventing silent breakages when prompts, models or RAG settings change: deterministic contract assertions, semantic equivalence checks, tool-call verification, snapshot diffs and release sign-off matrices.
- Live Course
- 5 modules
-
Deep observability for multi-step LLM, RAG and agent systems: OpenTelemetry GenAI conventions, self-hosted Langfuse and Arize Phoenix, per-span token/latency/cost breakdown, PII masking and production trace dashboards.
- Live Course
- 5 modules
-
Low-latency input and output guardrails for production LLM applications: blocking prompt injection and jailbreaks, KVKK/GDPR-compliant PII masking, topic rails, hallucination checks and safe tool execution controls.
- Live Course
- 5 modules
-
Cutting cost and latency in production LLM applications with multi-layer caching: provider and vLLM prompt caching, Redis exact caching, vector-based semantic caching, tenant isolation and cache invalidation.
- Live Course
- 5 modules
Get in touch
