Course Objective & Scope
As AI initiatives move from prototypes to production, cloud GPU bills and external LLM token spend can quickly outpace business value: forgotten interactive GPU notebooks, oversized inference instances running at 15% utilisation and routing every trivial query to a frontier LLM erode unit economics. This course combines AI FinOps with capacity engineering to control infrastructure spend without slowing teams down.
GPU utilisation telemetry (SM occupancy versus VRAM allocation), per-team showback/chargeback with OpenCost/Kubecost and AI Gateways, fractional GPU sharing (MIG/time-slicing), priority queuing with Kueue, fault-tolerant Spot training and tiered model routing are covered hands-on. Participants build an AI unit-cost model and GPU capacity optimisation plan for their organisation.
Who Should Attend?
- AI platform, MLOps and infrastructure engineers managing GPU clusters and cloud AI budgets
- FinOps practitioners and cloud architects optimising compute and AI service spend
- Engineering leaders accountable for the unit economics (cost per 1,000 predictions or per active user) of AI products
- Prerequisite: Familiarity with cloud computing, Kubernetes and ML/LLM training and inference workloads
Key Takeaways
- Establish an AI FinOps unit-economics framework tracking cost per 1,000 inferences and cost per resolved task
- Expose the gap between requested GPU capacity and actual SM/memory utilisation using NVIDIA DCGM and OpenCost/Kubecost
- Increase effective cluster throughput 2-3x via MIG, time-slicing, scale-to-zero and Kueue priority preemption
- Run long training jobs reliably on 60-70% cheaper Spot/Preemptible GPUs using automated checkpointing and recovery
- Cut GenAI token spend through semantic caching, prompt budgets and tiered routing between small models and frontier LLMs
Curriculum
AI Unit Economics and Multi-Tenant Cost Visibility
- Breaking down spend across training runs, 24/7 inference endpoints, storage/egress and external LLM APIs
- Implementing per-team, per-product and per-tenant showback/chargeback with OpenCost/Kubecost and AI Gateway logs
GPU Telemetry and Diagnosing Under-Utilisation
- Allocated GPUs versus active compute: reading DCGM SM occupancy, Tensor Core activity and VRAM high-water marks
- Right-sizing hardware tiers (L4, A10G, L40S, H100) to workload profiles (small CV models, LLM decode, distributed training)
Cluster Density: Fractional GPUs, Queuing and Autoscaling
- Bin-packing lightweight models and interactive notebooks onto shared GPUs with MIG and Time-Slicing
- Enforcing team quotas, cohort capacity borrowing, off-peak batch preemption with Kueue and scale-to-zero endpoints
Spot/Preemptible GPU Execution and Capacity Commitment Planning
- Handling spot eviction notices gracefully with fast asynchronous checkpointing and automatic job resumption
- Blending Reserved/Savings Plans or on-prem base capacity with elastic Spot and On-Demand burst pools
GenAI / LLM Token FinOps and Budget Guardrails
- Tiered model routing (cascading from small open models to frontier LLMs), async Batch APIs and prefix/semantic caching
- Enforcing hard daily/monthly token spend caps per tenant, anomaly detection on runaway agents and cost-vs-quality dashboards
Other Courses in This Category
-
Moving from Jupyter notebooks to production: modular Python code, DVC, MLflow, FastAPI packaging, Docker and GitHub Actions, for data scientists and engineers building a working end-to-end machine learning pipeline.
- Live Course
- 5 modules
-
MLOps Fundamentals
MLOps training covering machine learning model lifecycle management, containerization with Docker and Kubernetes, CI/CD automation with MLflow/Kubeflow, and model drift monitoring.
- Live Course
- 6 modules
-
Systematic experiment design, hyperparameter optimisation with Optuna and Ray Tune, metric/artefact standards, reproducibility and comparative setup of tracking tools (MLflow, W&B, Neptune), for data science teams.
- Live Course
- 5 modules
-
MLOps – Advanced Level
Enterprise-grade MLOps covering model drift monitoring, A/B testing, canary deployments, automated retraining, CI/CD pipelines, Kafka/Spark Streaming, and Kubernetes orchestration.
- Live Course
- 6 modules
-
MLflow Tracking Server architecture, PostgreSQL/S3 setup, autologging, custom model packaging with pyfunc, MLflow Evaluate (classical ML and LLMs) and Model Registry integration, for teams standardising on open-source MLflow.
- Live Course
- 5 modules
-
W&B Experiments, Sweeps, Artifacts (data and model lineage), visual error analysis with Tables, LLM tracing with Weave and collaborative Reports, for deep learning and generative AI teams.
- Live Course
- 5 modules
Get in touch
