Course Objective & Scope
Many teams test their AI assistant simply by typing a handful of favourite prompts and eyeballing the output ('vibe checks')—an approach that feels fast at first but quickly erodes confidence whenever a prompt or model changes. This beginner-friendly programme builds practical LLM evaluation habits that teams can put to work immediately without heavy statistics or complex infrastructure.
Participants assemble a 25–30 question starter test set from their own domain, write simple deterministic checks alongside a clear pass/fail LLM-as-a-judge rubric, and run a side-by-side comparison of two prompts or models in Promptfoo.
Who Should Attend?
- Software developers getting started with LLM application development
- Product managers, business analysts and QA specialists defining quality criteria
- Technical leads establishing a shared evaluation culture across their team
- Prerequisite: basic familiarity with LLMs and prompting; advanced coding is not required
Key Takeaways
- Explain why ad-hoc vibe checks fall short and how the core eval loop works
- Curate a starter evaluation dataset from real user questions and failure examples
- Define fast code-based checks for JSON validity, required terms and length bounds
- Write a clear pass/fail LLM-as-a-judge rubric for subjective quality criteria
- Compare two prompts or models side by side in Promptfoo and produce a decision report
Curriculum
Why Evals? Moving Beyond Vibe Checks
- Why 'it worked yesterday' happens whenever prompts or models change
- The three pillars of practical evals: test cases, assertions and fast feedback
Building Your First Evaluation Dataset
- Balancing typical questions, tricky edge cases and out-of-scope requests
- Structuring inputs, expected facts and context columns in CSV, YAML or JSONL
Fast Deterministic Rule-Based Checks
- Contains/not-contains, regex, valid JSON and response length assertions
- Catching obvious failures instantly at zero token cost
Introduction to LLM-as-a-Judge
- How to use an LLM to grade open-ended responses reliably
- Writing crisp PASS/FAIL rubrics instead of vague 1-to-10 rating scales
Hands-On Comparison Lab with Promptfoo
- Running two prompt or model variants from the CLI and web matrix view
- Interpreting pass rates, failure cases and cost tables to make a release decision
Other Courses in This Category
-
Designing high-quality golden datasets that anchor LLM evaluations: scenario matrices, synthetic test generation and filtering from documents, adversarial/edge-case design, expert annotation protocols and dataset maintenance.
- Live Course
- 5 modules
-
Hybrid evaluator design combining deterministic rule-based assertions and model-graded (LLM-as-a-judge) scoring: regex, AST and JSON schema checks, pairwise and rubric-based judge prompts, bias control and calibration.
- Live Course
- 5 modules
-
Component-level and end-to-end evaluation for RAG pipelines and multi-step agents: retrieval (precision@k, recall@k, MRR), generation (faithfulness, relevance) and agent metrics (tool selection, argument accuracy, trajectory efficiency).
- Live Course
- 5 modules
-
Connecting pre-release offline eval scores with live production outcomes: offline-to-online correlation, shadow deployments, A/B testing and interleaving for LLM responses, business metric alignment and staged rollouts.
- Live Course
- 5 modules
-
Turning user interactions into continuous improvement in production LLM products: explicit and implicit feedback capture, trace-linked scoring, expert correction queues and building an eval and fine-tuning data flywheel.
- Live Course
- 5 modules
-
Fundamentals of moving prototype LLM applications into production: how LLMOps differs from classic MLOps, prompt and model versioning, basic tracing, starter eval datasets, guardrails and a production readiness checklist.
- Live Course
- 5 modules
Get in touch
