Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

In classic SRE incidents a server crashes and returns HTTP 500; in AI systems the service can return HTTP 200 while promising unauthorized discounts, hallucinating from a corrupted RAG document or burning thousands of dollars in a runaway agent loop. This course teaches how to classify failure modes unique to LLM and agent systems and build structured incident response workflows.

Through realistic incident drills (jailbreak/data leak, provider outage, poisoned RAG index, runaway agent), participants design kill switches, graceful degradation modes and an AI post-mortem process that turns every incident into a permanent guardrail and regression test.

Who Should Attend?

  • SRE, LLMOps and platform engineers operating AI services in production
  • Software and AI engineers participating in on-call and incident response rotations
  • Technical managers responsible for operational risk, security and business continuity
  • Prerequisite: foundational production operations and LLM application architecture knowledge

Key Takeaways

  • Classify AI incidents by severity (SEV1–SEV4) across safety, quality, cost and availability dimensions
  • Engineer feature flags, kill switches and deterministic/search-only fallback modes
  • Quarantine poisoned RAG documents, bad prompt releases or unsafe tool permissions within minutes
  • Stop cost spikes and runaway agent loops automatically using token and step circuit breakers
  • Run blameless AI post-mortems that convert every incident into a new guardrail and regression test

Curriculum

AI-Specific Failure Modes and Severity Classification
  • Silent failures: HTTP 200 hallucinations, erroneous tool calls and policy/tone violations
  • SEV1–SEV4 classification matrix across safety/PII leaks, cost spikes, outages and quality drops
Early Detection, Anomaly Alerts and Circuit Breakers
  • Alerting on token-burn spikes, guardrail block surges, repeated tool errors and thumbs-down bursts
  • Per-session agent step/spend limits and automated circuit-breaker rules
Immediate Containment: Kill Switches and Graceful Degradation
  • Zero-deploy prompt/model rollback, disabling risky tools and quarantining bad RAG documents
  • Degrading gracefully to keyword search, static rules or human agent handoff
Scenario Runbooks and Incident Drills
  • Response playbooks for active prompt injection/jailbreak campaigns and PII exposure
  • Traffic failover during regional cloud API outages or internal GPU node failures
AI Post-Mortems and Building System Immunity
  • Reconstructing incident timelines from traces, prompt versions, retrieved chunks and model configs
  • Translating root causes into permanent regression tests and updated guardrail rules

Other Courses in This Category

  • Hybrid Cloud and On-Premises AI Deployment Architectures

    Combining managed cloud AI platforms (Azure AI Foundry, Amazon Bedrock, Google Vertex AI) with on-premises GPU clusters in a hybrid architecture: sensitivity-based routing, portable contracts, private networking and disaster recovery.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • LLM Evaluations (Evals) Fundamentals: A Practical Introduction

    Moving from ad-hoc 'vibe checks' to measurable quality in LLM applications: building a first eval dataset, rule-based assertions, basic LLM-as-a-judge grading and running a first automated evaluation report with Promptfoo.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • Golden Dataset Curation and Test Case Design for LLM Applications

    Designing high-quality golden datasets that anchor LLM evaluations: scenario matrices, synthetic test generation and filtering from documents, adversarial/edge-case design, expert annotation protocols and dataset maintenance.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • Rule-Based Assertions and Model-Graded (LLM-as-a-Judge) Evaluators

    Hybrid evaluator design combining deterministic rule-based assertions and model-graded (LLM-as-a-judge) scoring: regex, AST and JSON schema checks, pairwise and rubric-based judge prompts, bias control and calibration.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • Evaluating RAG Pipelines and Autonomous AI Agents

    Component-level and end-to-end evaluation for RAG pipelines and multi-step agents: retrieval (precision@k, recall@k, MRR), generation (faithfulness, relevance) and agent metrics (tool selection, argument accuracy, trajectory efficiency).

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • Bridging Offline Benchmarks and Online Production Evaluation for LLMs

    Connecting pre-release offline eval scores with live production outcomes: offline-to-online correlation, shadow deployments, A/B testing and interleaving for LLM responses, business metric alignment and staged rollouts.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View

Get in touch