Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

A candidate prompt or model that scores well on an offline test set can still hurt user satisfaction in production whenever static benchmarks fail to capture real interaction dynamics. This course teaches a unified decision framework that bridges pre-deployment offline evaluation with live online experimentation and measurement.

Participants analyse the correlation between offline proxy metrics and real user outcomes (acceptance rate, task completion, resolution rate), and validate candidate models safely in production using shadow deployments, feature-flagged A/B tests and staged rollouts.

Who Should Attend?

  • Data scientists and AI engineers running model and prompt experiments on LLM products
  • Technical product managers and analytics teams leading A/B testing and product evaluation
  • LLMOps and platform engineers building progressive delivery and experimentation infrastructure
  • Prerequisite: foundational LLM evaluation concepts and basic statistics familiarity

Key Takeaways

  • Measure correlation between offline evaluation metrics and live product/business KPIs
  • Evaluate candidate models on live traffic without user impact using shadow (dark) deployments
  • Design statistically sound A/B experiments for chat, RAG and coding/writing assistants
  • Combine online judge scores, implicit user signals and business metrics in one experiment scorecard
  • Close the offline-online gap by updating offline benchmarks when live experiments diverge

Curriculum

Connecting the Offline and Online Evaluation Loop
  • Complementary strengths of offline evals (fast, safe, repeatable) and online evals (real users, real distribution)
  • Why offline proxy scores can rise while live user satisfaction drops (Goodhart's Law)
Shadow (Dark) Deployments and Live Traffic Replay
  • Mirroring live requests asynchronously to a candidate model for pairwise comparison against production
  • Read-only shadow testing and handling side effects in tool-calling agents
Designing A/B Experiments for LLM Applications
  • User- and session-level randomisation, sample size estimation and novelty effects
  • Primary outcome metrics (task resolution, acceptance rate) and guardrail metrics (latency, cost, safety)
Explicit and Implicit Online Success Signals
  • Overcoming sparse thumbs-up/down data with copy events, edit distance, re-queries and escalation rates
  • Combining online LLM-as-a-judge grades with behavioural telemetry
Offline-to-Online Calibration and Release Governance
  • Root-cause analysis when offline winners underperform in live A/B tests
  • Feeding uncovered real-world patterns back into the offline golden dataset

Other Courses in This Category

  • User Feedback Loops and Data Flywheels for Production LLM Products

    Turning user interactions into continuous improvement in production LLM products: explicit and implicit feedback capture, trace-linked scoring, expert correction queues and building an eval and fine-tuning data flywheel.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • LLMOps Fundamentals: Operating LLM Applications in Production

    Fundamentals of moving prototype LLM applications into production: how LLMOps differs from classic MLOps, prompt and model versioning, basic tracing, starter eval datasets, guardrails and a production readiness checklist.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • End-to-End LLMOps: Production Architecture and Implementation

    Bringing full production discipline to an LLM application: LiteLLM gateway, Langfuse/Phoenix tracing, automated eval and regression tests, CI/CD quality gates, runtime guardrails and live user feedback loops.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • Versioning Prompts, Models, Tools and Datasets in LLM Systems

    Full reproducibility in LLM applications: prompt versioning with Langfuse/MLflow Prompt Registry and Git, model/LoRA and tool schema versioning, RAG/eval dataset snapshots and safe rollbacks.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • AI Model Gateways and Intelligent LLM Routing

    Consolidating cloud and self-hosted LLMs behind a single OpenAI-compatible gateway with LiteLLM and enterprise AI gateways: automated fallbacks, load balancing, complexity-based routing, virtual keys and budget/rate-limit governance.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • CI/CD Pipelines and Automated Quality Gates for AI and LLM Applications

    GitHub Actions and GitLab CI pipelines for LLM and RAG applications: fast PR smoke and eval suites, deterministic and model-graded quality gates, cost and latency budget checks, canary rollouts and automated rollbacks.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View

Get in touch