Course Objective & Scope
When a user reports that an assistant gave a wrong answer and took 18 seconds, standard application logs only show an HTTP 200 status; whether the failure happened during query rewriting, vector retrieval, reranking or a third tool call remains invisible. This course teaches how to instrument LLM and agent systems for end-to-end tracing and observability.
Participants instrument a multi-step application using OpenTelemetry GenAI semantic conventions, Langfuse and Arize Phoenix, and design an enterprise trace dashboard tracking sessions, users, latency, token spend and retrieval quality.
Who Should Attend?
- Software and AI engineers building RAG, agent and multi-step LLM workflows
- SRE and platform teams managing enterprise observability stacks (OTel, Grafana, Datadog, Langfuse)
- MLOps engineers diagnosing production AI bugs and performance bottlenecks
- Prerequisite: Python and foundational distributed systems/API development knowledge
Key Takeaways
- Structure session, trace and nested span hierarchies across multi-turn chats and agent loops
- Collect vendor-neutral telemetry using OpenTelemetry GenAI semantic conventions
- Deploy self-hosted Langfuse or Arize Phoenix and integrate it with LLM frameworks
- Scrub sensitive PII from prompts and completions before traces leave the application
- Build operational dashboards covering latency waterfalls, token cost, error rates and retrieval scores
Curriculum
LLM Observability Foundations: Sessions, Traces and Spans
- How GenAI traces differ from classic APM traces (high cardinality, large text payloads)
- Generation, retrieval, embedding, reranker, guardrail and tool span types
OpenTelemetry GenAI and Framework Instrumentation
- OTel GenAI semantic conventions, OpenInference and OpenLLMetry standards
- Instrumenting LangChain/LangGraph, LlamaIndex and direct OpenAI/vLLM SDK calls
Self-Hosted Langfuse and Arize Phoenix
- Deploying Langfuse and Phoenix on Docker and Kubernetes
- Enriching traces with user IDs, session IDs, release tags and custom metadata
Privacy, PII Redaction and Performance Overhead
- Async background batching so tracing never slows down user requests
- Client-side scrubbing of national IDs, card numbers, emails and secrets before export
Trace Dashboard Design and Root-Cause Analysis
- Using latency waterfalls to spot slow retrieval steps and runaway agent loops
- Designing a unified dashboard for cost, P95 latency, error rates and user feedback
Other Courses in This Category
-
Low-latency input and output guardrails for production LLM applications: blocking prompt injection and jailbreaks, KVKK/GDPR-compliant PII masking, topic rails, hallucination checks and safe tool execution controls.
- Live Course
- 5 modules
-
Cutting cost and latency in production LLM applications with multi-layer caching: provider and vLLM prompt caching, Redis exact caching, vector-based semantic caching, tenant isolation and cache invalidation.
- Live Course
- 5 modules
-
End-to-end latency management in multi-step LLM, RAG and agent applications: SSE streaming UX, parallel retrieval/guardrail/tool execution, output token reduction, P95/P99 tail-latency control and hedged request patterns.
- Live Course
- 5 modules
-
Incident management tailored to production LLM and agent systems: response runbooks for hallucination spikes, prompt injection/data leaks, runaway agent loops and provider outages, kill-switch controls and AI post-mortems.
- Live Course
- 5 modules
-
Combining managed cloud AI platforms (Azure AI Foundry, Amazon Bedrock, Google Vertex AI) with on-premises GPU clusters in a hybrid architecture: sensitivity-based routing, portable contracts, private networking and disaster recovery.
- Live Course
- 5 modules
-
Moving from ad-hoc 'vibe checks' to measurable quality in LLM applications: building a first eval dataset, rule-based assertions, basic LLM-as-a-judge grading and running a first automated evaluation report with Promptfoo.
- Live Course
- 5 modules
Get in touch
