Course Objective & Scope
As customer behaviour, inflation, seasonality, competitor promos or sensor conditions shift, a model that scored 95% during validation gradually degrades in production. Yet not every distribution change hurts model accuracy, and blindly retraining on every PSI spike wastes compute and destabilises pipelines. This advanced course covers the statistical detection of data and concept drift and how to respond effectively.
Distinguishing covariate shift, label shift and true concept drift; univariate (PSI, KS, Chi-Square, Jensen-Shannon, Wasserstein) and multivariate/embedding (PCA reconstruction error, MMD) tests; estimating accuracy drop without ground truth via Evidently and NannyML (CBPE/DLE); and SHAP-weighted drift alerting are all practised hands-on. Participants design a low-noise drift detection and retraining policy for their own models.
Who Should Attend?
- Senior data scientists analysing temporal decay and distribution shifts in production models
- MLOps engineers building automated drift evaluation jobs and Continuous Training triggers
- Model validation and risk specialists in finance, credit scoring, insurance, retail and demand forecasting
- Prerequisite: Python, solid probability/statistics fundamentals and experience with production ML models
Key Takeaways
- Distinguish mathematically between covariate shift, prior probability (label) shift and true concept drift
- Choose the right statistical test or distance metric for sample size and modality (continuous, categorical, text/image embeddings)
- Weight feature drift by SHAP importance so alerts fire only on shifts that actually move predictions
- Estimate expected ROC-AUC or RMSE degradation before ground-truth labels arrive using NannyML and Evidently
- Select the right remediation tactic—sliding-window retraining, sample reweighting or threshold recalibration—when drift hits
Curriculum
Taxonomy of Drift: Covariate, Label and Concept Shift
- Decomposing joint probability shifts across P(X), P(Y) and conditional P(Y|X)
- Sudden, gradual and seasonal shifts versus upstream data-engineering breakages
Statistical Drift Tests for Tabular Data and Common Pitfalls
- Kolmogorov-Smirnov (KS), Population Stability Index (PSI), Chi-Square, Jensen-Shannon and Wasserstein distance
- Why p-values over-fire on large samples, choosing reference windows and SHAP-weighted drift prioritisation
Multivariate and Unstructured Embedding Drift
- Catching broken feature correlations via PCA reconstruction error and adversarial domain classifiers
- Monitoring text (LLM/RAG) and vision embeddings with centroid cosine distance, Mahalanobis distance and MMD
Estimating Performance Without Ground Truth (Evidently & NannyML)
- Confidence-Based Performance Estimation (CBPE) from well-calibrated predicted probabilities
- Direct Loss Estimation (DLE) for regression models and filtering out benign covariate drift
Drift Remediation and Continuous Adaptation Strategies
- When retraining helps versus when it hurts: sliding windows, forgetting factors and regime changes
- Covariate importance weighting, probability threshold recalibration and active learning on drifted cohorts
Other Courses in This Category
-
Feature Store architectures preventing training-serving skew and time leakage: offline (Parquet/Snowflake/BigQuery) and online (Redis) stores with Feast, point-in-time joins and streaming features, for data and MLOps teams.
- Live Course
- 5 modules
-
Pre-deployment fairness (Fairlearn), explainability (SHAP), probability calibration, uncertainty quantification (Conformal Prediction), guardrails, progressive rollouts and kill-switches, for ML and risk teams.
- Live Course
- 5 modules
-
Designing an enterprise AI platform unifying classical ML and GenAI/LLMs: compute/GPU orchestration, data and model layers, AI Gateways, multi-tenant isolation and reference architectures, for architects and platform leads.
- Live Course
- 5 modules
-
Designing an Internal Developer Platform (IDP) for data scientists and AI engineers: Backstage templates, on-demand notebook/GPU workspaces, Golden Path pipelines and self-service deployment, for platform teams.
- Live Course
- 5 modules
-
IAM/RBAC/ABAC for AI workloads, secret management with HashiCorp Vault and External Secrets, Policy-as-Code with OPA/Kyverno, supply-chain security (pickle/images) and network isolation, for platform and security teams.
- Live Course
- 5 modules
-
GPU and LLM token unit economics, cost visibility with DCGM/Kubecost/OpenCost, quota/priority scheduling with Kueue, MIG/fractional GPU sharing, Spot training and model routing, for AI platform and FinOps leads.
- Live Course
- 5 modules
Get in touch
