Course Objective & Scope
Unlike evaluating a single-turn LLM response, an agent can reach the same goal via different tool sequences, recover from mid-step errors or mutate external system state. This course covers methods for measuring both an agent's final outcome and the trajectory it took to get there.
Participants build golden datasets, mock/sandbox tool environments, LLM-as-a-judge rubrics and deterministic state assertions using LangSmith, Braintrust or Promptfoo, delivering a case-specific eval suite and pre-release quality checklist.
Who Should Attend?
- AI and software engineers developing agents
- AI quality assurance (QA), testing and LLMOps teams
- Technical product managers accountable for agent performance
- Prerequisite: Python and basic LLM/agent development experience
Key Takeaways
- Define metrics across final output, execution trajectory and environment state
- Build deterministic mock tools and sandboxes for side-effect-free testing
- Automatically measure tool selection, argument accuracy and redundant steps
- Create user simulators for multi-turn agent evaluation
- Set up automated eval pipelines that catch regressions on model or prompt changes
Curriculum
Challenges of Agent Evaluation and Metric Framework
- Task success (pass@k), consistency (pass^k), step efficiency and cost
- Outcome-based versus trajectory-based evaluation
Golden Datasets and Mock Environment Design
- Mining test scenarios and edge cases from production traces
- Mock APIs and seeded database states for side-effect-free runs
Step and Trajectory Evaluation
- Tool selection accuracy, argument schema checks and ordering assertions
- Partial credit scoring and detecting redundant or repeated calls
Multi-Turn Simulation and LLM-as-a-Judge
- Testing end-to-end dialogues with persona-driven user simulators
- Calibrating judge models against human annotations
CI/CD Integration and Online Evaluation
- Automated regression gates on prompt, tool or model updates
- Sampling live production traces (online evals) and release checklist
Other Courses in This Category
-
Securing tool-using agents that act on external systems: indirect prompt injection defence, delegated user authorisation with OAuth/OBO, sandbox isolation, guardrails and OWASP Agentic security controls.
- Live Course
- 5 modules
-
Foundational programme on how LLM-based agents work: model, tools, memory and the control loop; how agents differ from chatbots and workflow automation, and how to pick the right first use case for your organisation.
- Live Course
- 6 modules
-
Agentic AI for executives: impact on business models, processes and organisation, function-by-function use cases, investment and ROI assessment, operational risk, governance and redesigning teams that work alongside agents.
- Live Course
- 5 modules
-
Hands-on programme on how LLMs use APIs, databases and internal systems via function calling / tool calling: tool schema design, argument validation, error handling and a working agent built on OpenAI, Claude and Gemini APIs.
- Live Course
- 6 modules
-
Method for turning business processes into agentic workflows: separating deterministic steps from LLM decisions, prompt chaining, routing, parallelisation and evaluator–optimizer patterns, implemented in n8n or Power Automate.
- Live Course
- 6 modules
-
Developer programme on building autonomous agents with LangGraph, the OpenAI Agents SDK and the Claude Agent SDK: state management, tool integration, loop control, tracing and a deployment-ready agent for an organisational case.
- Live Course
- 6 modules
Get in touch
