Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

In multi-model pipelines, RAG systems or agentic workflows where CPU preprocessing and GPU inference steps call one another, scaling components natively in Python avoids microservice sprawl. Ray Serve provides a Python-native distributed serving layer on top of Ray clusters with fractional GPU allocation, independent replica autoscaling and built-in vLLM integration.

Participants compose multi-model graphs using @serve.deployment and DeploymentHandle, enable dynamic batching with @serve.batch, integrate FastAPI ingress routes and deploy multi-node services via KubeRay. Teams leave with a working Ray Serve application and declarative serve config tailored to their use case.

Who Should Attend?

  • ML and AI engineers scaling multi-model pipelines, RAG and multi-agent architectures
  • MLOps and platform engineers operating GPU clusters on Ray and Kubernetes (KubeRay)
  • Developers who want distributed model serving without leaving Python
  • Prerequisite: intermediate Python, async/await familiarity and basic model serving knowledge

Key Takeaways

  • Allocate CPU and GPU resources—including fractional GPUs—per model with @serve.deployment
  • Chain multiple models and business logic asynchronously using DeploymentHandle
  • Adapt to traffic spikes using @serve.batch and per-deployment replica autoscaling
  • Stand up OpenAI-compatible scalable LLM endpoints using Ray Serve LLM and vLLM
  • Perform zero-downtime production rollouts with serve build and KubeRay RayService

Curriculum

Ray Architecture and Ray Serve Fundamentals
  • Ray actor model, Head/Worker nodes, HTTP Proxy and Replica architecture
  • @serve.deployment, FastAPI (@serve.ingress) integration and building a first service
Resource Allocation, Dynamic Batching and Autoscaling
  • CPU and fractional GPU (num_gpus=0.25) sharing via ray_actor_options
  • The @serve.batch decorator and queue-driven autoscaling_config settings
Multi-Model Composition and Model Multiplexing
  • Chaining preprocessing, classifier and reranker deployments with DeploymentHandle
  • Serving hundreds of LoRA adapters or tenant models with LRU caching via @serve.multiplexed
Serving Large Language Models with Ray Serve and vLLM
  • Integrating the vLLM engine, tensor/pipeline parallelism and placement groups
  • Streaming responses, multi-LoRA serving and an OpenAI-compatible API layer
Production Deployment with KubeRay and Observability
  • Generating declarative YAML with serve build and zero-downtime updates via KubeRay RayService
  • Ray Dashboard, Prometheus metrics and distributed log/trace inspection

Other Courses in This Category

  • Model Versioning, Rollout Strategies and Safe Replacement

    Updating production models safely without downtime: MLflow Model Registry, semantic versioning, shadow deployments, canary rollouts, blue/green switches, A/B testing and automated rollback. For MLOps and ML engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Scaling Deployed Machine Learning and LLM Services

    Horizontal and vertical scaling for live model services: autoscaling on queue/GPU metrics with Kubernetes HPA, KEDA and Knative, reducing cold starts, MIG/time-slicing and load balancing. For MLOps, SRE and platform engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Introduction to Model Deployment

    The core steps for moving a trained model from notebook to production: serialization, serving via an API, container packaging, basic monitoring and rollback. For data scientists and developers new to model deployment.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • REST API and gRPC Fundamentals for Model Serving

    Fundamentals of REST and gRPC for model services: HTTP/JSON design, OpenAPI, Protocol Buffers, streaming RPCs and performance comparison. For developers who want to pick the right protocol and build their first service.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Synchronous vs Asynchronous Inference

    Synchronous and asynchronous inference patterns: request-response, queue-based processing (Celery, Redis, Kafka), job polling, webhooks and async Python. For engineers designing long-running model calls that scale.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Batch, Real-Time and Streaming Inference Fundamentals

    A comparison of the three core inference modes: batch scoring with Spark and Airflow, REST-based real-time services and streaming inference with Kafka/Flink. For data and ML teams choosing the right mode for a business need.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View

Get in touch