Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

The behaviour of an LLM application depends not only on application code, but on the system prompt, the exact model snapshot, tool/function JSON schemas, the embedding model and the state of the knowledge index. When any one of those changes silently, reproducing a production bug becomes impossible. This course teaches how to manage all of these artefacts together as a traceable release bundle.

In hands-on labs participants set up Langfuse and MLflow Prompt Registry, Git-based prompt-as-code workflows, JSON Schema evolution and dataset versioning with DVC/LakeFS, building a single-command rollback capability.

Who Should Attend?

  • Software and AI engineers building LLM and agent applications
  • MLOps and LLMOps engineers managing model, data and configuration lifecycles
  • Technical teams in regulated industries requiring auditability and version lineage
  • Prerequisite: Git proficiency, Python and foundational LLM application development

Key Takeaways

  • Version prompt templates alongside variables, model parameters and environment labels (`dev`, `staging`, `prod`)
  • Track model snapshots, LoRA adapter versions and embedding indices together
  • Maintain backward compatibility when evolving tool and function-calling schemas
  • Pin eval datasets and RAG document corpora using hashes and data versioning tools
  • Link every production trace to the exact prompt, model, tool and data version that produced it

Curriculum

Why Git Alone Is Not Enough for LLM Systems
  • Non-code artefacts: prompts, model weights/snapshots, tool schemas and vector indices
  • Unified release manifests and end-to-end reproducibility
Prompt Registries and Prompt-as-Code Workflows
  • Runtime cached prompt fetching with Langfuse and MLflow Prompt Registry
  • Git-backed YAML/Jinja templates, schema validation and environment labels (`staging`/`prod`)
Model and Tool Schema Versioning
  • API model date snapshots, local weight digests and LoRA adapter registries
  • Version evolution and contract testing for agent tool/function JSON schemas
Eval Dataset and RAG Index Versioning
  • Dataset snapshots with DVC, LakeFS or Hugging Face Datasets
  • Blue/green vector index migration when embedding models or chunking rules change
Lineage, Audit Trails and Instant Rollback
  • Attaching prompt_version, model_id and index_version metadata to every trace
  • Rolling back prompt and model versions in production without redeploying code

Other Courses in This Category

  • AI Model Gateways and Intelligent LLM Routing

    Consolidating cloud and self-hosted LLMs behind a single OpenAI-compatible gateway with LiteLLM and enterprise AI gateways: automated fallbacks, load balancing, complexity-based routing, virtual keys and budget/rate-limit governance.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • CI/CD Pipelines and Automated Quality Gates for AI and LLM Applications

    GitHub Actions and GitLab CI pipelines for LLM and RAG applications: fast PR smoke and eval suites, deterministic and model-graded quality gates, cost and latency budget checks, canary rollouts and automated rollbacks.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • LLM Evaluation Methodology, Metrics and Frameworks

    Systematic quality measurement for LLM applications: deterministic and LLM-as-a-judge metrics with Promptfoo, DeepEval and Ragas, mitigating judge biases, calibrating against human labels and building an enterprise eval scorecard.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • Engineering Automated and Scalable LLM Evaluation Pipelines

    Engineering fast, repeatable, large-scale LLM evaluation pipelines: async parallel test runners, judge cost and cache optimisation, dataset ingestion, statistical significance and experiment comparison dashboards.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • Continuous Online Evaluation and Drift Detection for Production LLMs

    Continuous quality monitoring on live LLM traffic: smart trace sampling, zero-latency async online evaluators, reference-free hallucination and relevance scoring, query drift detection and automated alerting.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View
  • Regression Testing for LLM, RAG and Agent Applications

    Preventing silent breakages when prompts, models or RAG settings change: deterministic contract assertions, semantic equivalence checks, tool-call verification, snapshot diffs and release sign-off matrices.

    LLMOps & AI in Production Corporate
    Live Course
    5 modules
    View

Get in touch