Course Objective & Scope
As the number of models and traffic grow, a single Flask service no longer suffices; the right architecture determines cost, reliability and team velocity. The programme compares embedded model, model-as-a-service, serverless, edge and multi-model serving patterns, as well as inference graphs and LLM gateway designs.
Platforms such as KServe, NVIDIA Triton, Ray Serve, BentoML and vLLM are positioned by the problems they solve, and Vertex AI, Amazon SageMaker and Azure AI Foundry are reviewed as managed options. Participants draft a target architecture and architecture decision records for one of the organisation's scenarios.
Who Should Attend?
- Solution and software architects designing an AI platform
- Senior ML and MLOps engineers
- Platform and infrastructure team leads
- Prerequisite: model deployment and container fundamentals
Key Takeaways
- Compare the main model serving patterns with their pros and cons
- Select and justify a serving platform for a scenario
- Design multi-model and inference graph structures
- Plan gateway, cache and routing layers for LLM workloads
- Document the target architecture with architecture decision records
Curriculum
Core Serving Patterns
- Embedded model and model-as-a-service
- Serverless and edge deployment
Multi-Model Serving and Inference Graphs
- Shared model servers and dynamic loading
- Pre-processing, model and post-processing chains
Positioning Serving Platforms
- KServe, Triton, Ray Serve, BentoML
- Managed services: Vertex AI, SageMaker, Azure AI Foundry
LLM Serving Architectures
- Self-hosted serving with vLLM and SGLang
- LLM gateway, semantic cache and model routing
Target Architecture Workshop
- An architecture draft for an organisational scenario
- Architecture decision records and risk analysis
Other Courses in This Category
-
Balancing p95/p99 latency, throughput and unit cost in model serving: dynamic batching, quantization, caching, GPU/CPU selection and load-testing optimisation. For ML and platform engineers building scalable, cost-effective inference infrastructure.
- Live Course
- 5 modules
-
End-to-end security for inference services: mTLS, OAuth2/JWT, rate limiting, safe serialization, container hardening, defence against model extraction/adversarial attacks and data privacy. For ML, platform and AppSec engineers.
- Live Course
- 5 modules
-
Exporting TensorFlow and Keras models as SavedModel, defining SignatureDefs and serving them over REST and gRPC with TensorFlow Serving. For data scientists and ML engineers starting to deploy TensorFlow models.
- Live Course
- 5 modules
-
Multi-model configurations, dynamic batching, XLA/TensorRT and mixed-precision acceleration, GPU memory tuning, Prometheus metrics and custom servables in TensorFlow Serving. For engineers operating high-traffic TF services.
- Live Course
- 5 modules
-
Operating TensorFlow Serving at scale on Docker and Kubernetes: pulling models from object storage, GPU scheduling, readiness/liveness probes, HPA/KEDA autoscaling and KServe integration. For MLOps and platform engineers.
- Live Course
- 5 modules
-
Deploying PyTorch models with TorchServe: writing custom Handlers, packaging .mar files with torch-model-archiver, using the Inference and Management APIs, and managing workers and versions. For data scientists and ML engineers using PyTorch.
- Live Course
- 5 modules
Get in touch
