Course Objective & Scope
Data science teams routinely run hundreds of training trials, yet the exact dataset split, Git commit, hyperparameter set and random seed behind the winning model often get lost in spreadsheets or personal notes. Rather than focusing on a single vendor tool, this course teaches enterprise experiment-management methodology, strict reproducibility and automated hyperparameter optimisation (HPO).
Bayesian search and early-stopping (pruning) strategies with Optuna and Ray Tune, naming and tagging taxonomies, error-slice analysis and an architectural comparison of tracking platforms (MLflow, Weights & Biases, Comet/Neptune) are practised hands-on. Participants leave with a team-wide experiment tracking protocol and reusable HPO template.
Who Should Attend?
- Data scientists running frequent model experiments and hyperparameter sweeps
- ML engineers establishing shared experiment standards across multiple projects
- Team leads responsible for the velocity and auditability of model development
- Prerequisite: Experience training ML models in Python (scikit-learn, LightGBM or PyTorch)
Key Takeaways
- Build reproducible experiment templates that automatically capture code commits, data hashes, environment and hardware info
- Define a team-wide experiment taxonomy covering project, task, dataset version and candidate-model tags
- Run TPE/Bayesian hyperparameter searches and prune unpromising trials early (ASHA/Hyperband) with Optuna and Ray Tune
- Compare candidate models across data slices, calibration curves and latency rather than a single aggregate score
- Select and architect the right tracking stack (MLflow, W&B or alternatives) for on-prem security, scale and cost constraints
Curriculum
Reproducibility and Experiment Taxonomy
- Deterministic training: random seeds, Git commit SHAs, dataset digests and dependency locking
- Enterprise naming conventions, tagging standards and the discipline of establishing baselines
What to Log: Metrics, Artefacts and Error Slices
- Training/validation curves, gradient norms, system utilisation and inference latency
- Logging confusion matrices, PR/ROC curves, SHAP summaries and worst-performing hard examples
Intelligent Hyperparameter Optimisation with Optuna
- Moving beyond grid/random search to TPE and multi-objective optimisation (accuracy vs latency)
- Saving GPU hours by terminating unpromising trials early with MedianPruner and Hyperband
Distributed HPO and Experiment Orchestration (Ray Tune & Hydra)
- Running parallel sweeps and managing compute quotas with Hydra multirun and Ray Tune
- Streaming trial metrics to tracking servers and analysing results via parallel-coordinate plots
Comparing Tracking Platforms and Enterprise Governance
- MLflow, Weights & Biases and Neptune/ClearML: architecture, self-hosted versus SaaS trade-offs and cost
- Access control, artefact retention policies and running structured model-review sessions
Other Courses in This Category
-
MLOps – Advanced Level
Enterprise-grade MLOps covering model drift monitoring, A/B testing, canary deployments, automated retraining, CI/CD pipelines, Kafka/Spark Streaming, and Kubernetes orchestration.
- Live Course
- 6 modules
-
MLflow Tracking Server architecture, PostgreSQL/S3 setup, autologging, custom model packaging with pyfunc, MLflow Evaluate (classical ML and LLMs) and Model Registry integration, for teams standardising on open-source MLflow.
- Live Course
- 5 modules
-
W&B Experiments, Sweeps, Artifacts (data and model lineage), visual error analysis with Tables, LLM tracing with Weave and collaborative Reports, for deep learning and generative AI teams.
- Live Course
- 5 modules
-
Synchronised versioning of large datasets, labels and model weights alongside Git using DVC, LakeFS and Iceberg/Delta time-travel, with end-to-end lineage for teams building auditable ML systems.
- Live Course
- 5 modules
-
Central Model Registry architecture, packaging standards, @champion/@challenger promotions, automated validation gates, model cards and security scanning, for teams governing how models move into production.
- Live Course
- 5 modules
-
Online, batch and streaming inference patterns, Triton Inference Server, BentoML, Ray Serve and vLLM, dynamic batching, gRPC/REST, caching and load testing, for teams serving models at low latency and high throughput.
- Live Course
- 5 modules
Get in touch
