Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

In traditional software passing unit tests is enough to ship, whereas in an LLM application a two-word prompt tweak or a new chunking parameter can quietly degrade answer quality without raising a syntax error. Conversely, running hundreds of live LLM calls on every commit slows CI to a crawl and burns budget. This course teaches how to design multi-stage AI CI/CD pipelines that balance speed, cost and rigour.

Using GitHub Actions or GitLab CI, participants build a pipeline that runs fast contract and subset evals on pull requests, executes full regression suites before merge and automatically rolls back canary deployments when production metrics degrade.

Who Should Attend?

  • DevOps, MLOps and platform engineers deploying LLM applications
  • Software engineers adapting continuous delivery workflows for AI projects
  • Technical team leads responsible for release quality and deployment automation
  • Prerequisite: Git, GitHub Actions or GitLab CI basics and Python familiarity

Key Takeaways

  • Separate fast unit/contract checks from costly LLM evaluations in a tiered CI pipeline
  • Post automated quality, latency and cost diff reports directly onto pull requests
  • Enforce blocking quality gates based on safety (prompt injection/PII) and accuracy thresholds
  • Roll out new prompt and model versions gradually using canary and shadow deployments
  • Trigger automated rollbacks when live operational or quality metrics breach thresholds

Curriculum

Tiered CI/CD Design for AI Applications
  • Deciding which checks belong on commit, PR, staging and nightly schedules
  • Balancing mocked/recorded responses (VCR cassettes) with live LLM calls
Static Checks, Contract Tests and Security Scans
  • Prompt template syntax, variable linting and tool JSON Schema validation
  • Automated prompt injection and vulnerability scanning in CI (Promptfoo/Garak)
PR Quality Gates and Diff Reporting
  • Running automated smoke evaluations against a golden dataset subset on PRs
  • Posting accuracy, TTFT and token cost diffs as pull-request comments
Progressive Delivery: Canary, Blue/Green and Shadow
  • Routing 5% of live traffic to a new prompt or model revision
  • Testing candidate models on live traffic in shadow mode without impacting users
Post-Deploy Verification and Automated Rollback
  • Monitoring error rates, latency and guardrail triggers immediately after rollout
  • Automatic rollback to the previous stable version via GitOps or gateway rules

Other Courses in This Category

  • LLM Evaluation Methodology, Metrics and Frameworks

    Systematic quality measurement for LLM applications: deterministic and LLM-as-a-judge metrics with Promptfoo, DeepEval and Ragas, mitigating judge biases, calibrating against human labels and building an enterprise eval scorecard.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • Engineering Automated and Scalable LLM Evaluation Pipelines

    Engineering fast, repeatable, large-scale LLM evaluation pipelines: async parallel test runners, judge cost and cache optimisation, dataset ingestion, statistical significance and experiment comparison dashboards.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • Continuous Online Evaluation and Drift Detection for Production LLMs

    Continuous quality monitoring on live LLM traffic: smart trace sampling, zero-latency async online evaluators, reference-free hallucination and relevance scoring, query drift detection and automated alerting.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • Regression Testing for LLM, RAG and Agent Applications

    Preventing silent breakages when prompts, models or RAG settings change: deterministic contract assertions, semantic equivalence checks, tool-call verification, snapshot diffs and release sign-off matrices.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • LLM Tracing and Observability with OpenTelemetry, Langfuse and Arize Phoenix

    Deep observability for multi-step LLM, RAG and agent systems: OpenTelemetry GenAI conventions, self-hosted Langfuse and Arize Phoenix, per-span token/latency/cost breakdown, PII masking and production trace dashboards.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • Runtime Guardrails and Safety Layers for Production LLM Applications

    Low-latency input and output guardrails for production LLM applications: blocking prompt injection and jailbreaks, KVKK/GDPR-compliant PII masking, topic rails, hallucination checks and safe tool execution controls.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View

Get in touch