Course Objective & Scope
Multi-step agents can reach the same answer through different paths—or call the right tool with wrong parameters and fail silently—making single-turn LLM metrics inadequate. This course teaches how to evaluate and monitor agent quality both pre-deployment (offline evals) and in production (online monitoring).
Participants practise end-to-end outcome evaluation, trajectory and single-step tool-call grading, deterministic mocked environments, LLM-as-a-judge rubrics and OpenTelemetry-based tracing in Langfuse, Arize Phoenix or LangSmith. The deliverable is an automated eval pipeline and a production observability dashboard for an organisational agent.
Who Should Attend?
- AI and software engineers preparing agent systems for production
- Data scientists and QA engineers validating model and agent quality
- Platform teams managing LLMOps and observability infrastructure
- Prerequisite: Python and experience building tool-using agents
Key Takeaways
- Write distinct evaluations for end-to-end task completion, trajectory efficiency and single-step decisions
- Mock side-effecting tools to create reproducible, deterministic test environments
- Instrument model calls, tool latencies and token costs with OpenTelemetry spans
- Automatically capture failing production traces and promote them into golden datasets
- Run regression gates in CI/CD whenever prompts, models or tool schemas change
Curriculum
The Three Levels of Agent Evaluation
- Final response (outcome), trajectory (path taken) and single-step evaluation
- Pass@k versus pass^k metrics: measuring consistency and reliability
Trajectory and Tool-Call Evals
- Expected tool sequences: exact match, in-order and any-order comparisons
- Tool selection accuracy, parameter schema compliance and redundant-step penalties
- Isolated test scenarios with mocked tools and state snapshots
Distributed Tracing and OpenTelemetry
- Run, trace and span hierarchy: LLM, tool, retriever and sub-agent spans
- OpenTelemetry GenAI semantic conventions with Langfuse, Phoenix and LangSmith
- PII redaction and privacy-preserving trace logging
Live Production Monitoring and Failure Triage
- Dashboards for token cost, latency, loop rate and tool error rate
- Running online evaluators on sampled production traffic
- Clustering failure modes and feeding production edge cases back into datasets
Lab: CI/CD Eval Gate and Observability Setup
- Building a golden dataset and automated regression suite for an enterprise agent
- Bottleneck and root-cause analysis on real agent traces
Other Courses in This Category
-
Training on protecting multi-step agents against API outages, rate limits, partial failures and crashes using retries, model fallbacks, idempotency, compensating actions and durable execution with Temporal and LangGraph.
- Live Course
- 5 modules
-
Architectural design course examining prompt chaining, routing, parallelisation, orchestrator-workers, evaluator-optimizer and autonomous agent patterns, and how to pick the simplest effective pattern for each problem.
- Live Course
- 5 modules
-
Hands-on developer training on combining ReAct reasoning loops, Plan-and-Execute task planning and explicit state/checkpoint management into a single unified hybrid agent architecture.
- Live Course
- 5 modules
-
Technical course on delegating work across specialist agents using control handoffs versus the agent-as-tool pattern, covering triage routing, context filtering and conversation ownership.
- Live Course
- 5 modules
-
Course on designing reusable skill packages, progressive context disclosure, bundled scripts/references and dynamic capability loading so agents scale without overflowing their context window.
- Live Course
- 5 modules
-
Course on cross-framework and cross-vendor agent collaboration using the A2A (Agent2Agent) protocol: Agent Cards, task lifecycle, streaming updates, enterprise authentication and MCP–A2A complementarity.
- Live Course
- 5 modules
Get in touch
