Course Objective & Scope
In traditional software testing, identical inputs yield identical outputs and exact assertions suffice; in LLM, RAG and agent systems, responses are probabilistic and a minor prompt or model tweak can trigger silent regressions. This course equips QA and test engineers with systematic methods to verify AI applications.
Curating golden datasets and synthetic edge cases, combining deterministic assertions with RAG triad metrics (faithfulness, context precision/recall, answer relevance), calibrating LLM-as-a-Judge graders, testing multi-step agent tool trajectories, and wiring Promptfoo/DeepEval/Ragas into CI quality gates are practised hands-on.
Who Should Attend?
- QA engineers, test automation specialists and SDETs
- Quality assurance leads and test architects
- Teams validating internal or customer-facing LLM, chatbot, RAG or AI agent products
- Prerequisite: experience with software testing methodologies and basic test automation
Key Takeaways
- Define testable acceptance criteria and statistical pass thresholds for non-deterministic AI features
- Build golden test datasets covering happy paths, domain edge cases and adversarial inputs
- Isolate whether RAG defects stem from retrieval misses or generation hallucinations
- Calibrate LLM-as-a-Judge rubrics against human annotations for reliable automated grading
- Run automated hallucination, safety and latency regression suites on every PR with Promptfoo or DeepEval
Curriculum
The Testing Paradigm for Non-Deterministic Systems
- Why exact string matching fails; layering deterministic, semantic and rubric-based checks
- The AI test pyramid: prompt/component unit checks, RAG/tool integration and multi-turn E2E tests
Golden Datasets and Synthetic Test Case Generation
- Designing happy-path, missing-context, ambiguous, multilingual and multi-turn test cases
- Harvesting anonymised production failures into living regression datasets
Testing RAG and Knowledge-Retrieval Systems
- Retrieval metrics (Hit Rate, MRR, Context Precision/Recall) and diagnosing chunking defects
- Generation metrics: faithfulness/groundedness, citation accuracy and abstention ('I don't know') checks
Agent Trajectory, LLM-as-a-Judge and Adversarial Safety Testing
- Verifying tool selection, argument schemas and multi-step loop limits in agents
- OWASP LLM Top 10 checks: prompt injection, PII leakage, jailbreaks and off-topic guardrails
CI/CD Test Automation and Release Sign-Off with Promptfoo / DeepEval
- Running side-by-side regression matrices whenever prompts, models or parameters change
- Release sign-off gates combining accuracy scores, p95 latency and token cost budgets
Other Courses in This Category
-
Training for SRE, DevOps and IT operations teams on alert noise reduction, log/metric/OpenTelemetry trace correlation, recent-change analysis, human-approved runbook execution and post-mortem automation.
- Live Course
- 5 modules
-
Accelerated transition programme taking software teams from basic IDE autocomplete and chat to context engineering, multi-file editing and test-verified autonomous coding agent workflows.
- Live Course
- 4 modules
-
Hands-on training on designing modular, path-scoped and testable AGENTS.md, CLAUDE.md, .cursor/rules and .github/copilot-instructions.md files so coding agents consistently follow team architecture and standards.
- Live Course
- 5 modules
-
Training on making structured specifications (SPEC.md, OpenAPI, JSON Schema, TypeSpec, state machines and executable acceptance tests) the primary engineering artifact that drives coding agents deterministically.
- Live Course
- 5 modules
-
Practical playbook for executing the three core engineering workloads with coding agents: end-to-end multi-file feature delivery, large-scale refactoring and framework/SDK/database migrations combining codemods with agent loops.
- Live Course
- 5 modules
-
Hands-on training in autonomous QA workflows using Playwright MCP, browser-driving testing agents, self-healing E2E tests and autonomous exploratory testing across UI and API user journeys.
- Live Course
- 5 modules
Get in touch
