Course Objective & Scope
A classifier that answers in milliseconds cannot be served the same way as a document analysis job or LLM call that takes minutes. The course covers the limits of synchronous request-response, asynchronous job patterns (queues, workers, polling by job ID, webhook callbacks) and writing asyncio-based services in Python.
In the labs participants build an inference queue with Celery and Redis, define timeout, retry and idempotency rules, and evaluate managed options such as Amazon SageMaker Asynchronous Inference. The result is a pattern selection checklist and a working reference implementation for one of the organisation's scenarios.
Who Should Attend?
- AI and backend engineers designing model services
- Teams running long LLM or document-processing workflows
- Platform engineers operating queue and worker infrastructure
- Prerequisite: Python and basic REST API knowledge
Key Takeaways
- Choose and justify a synchronous or asynchronous pattern for an inference workload
- Write a non-blocking model service with asyncio
- Build a queue-based inference flow with Celery/Redis
- Design result delivery via polling and webhooks
- Apply timeout, retry and idempotency rules
Curriculum
Synchronous Inference and Its Limits
- Request-response model, timeouts and connection limits
- Which workloads should stay synchronous?
asyncio and Async Services in Python
- Event loop, async/await and blocking-code pitfalls
- Async FastAPI endpoints and thread pool usage
Queue-Based Asynchronous Inference
- Job queues with Celery, Redis and Kafka
- Worker scaling and prioritisation
Result Delivery: Polling, Webhooks and Streaming
- Status queries by job ID
- Webhook callbacks and server-sent events
Reliability and Managed Options
- Retries, idempotency and dead-letter queues
- SageMaker Asynchronous Inference and similar services
Other Courses in This Category
-
A comparison of the three core inference modes: batch scoring with Spark and Airflow, REST-based real-time services and streaming inference with Kafka/Flink. For data and ML teams choosing the right mode for a business need.
- Live Course
- 5 modules
-
Saving models safely and portably: pickle/joblib risks, TorchScript, SavedModel, ONNX, safetensors and GGUF formats, packaging with MLflow and version compatibility. For ML engineers and data scientists.
- Live Course
- 5 modules
-
Model serving architecture patterns: embedded models, model-as-a-service, serverless, edge, multi-model and LLM gateway designs, and how platforms like KServe, Triton, Ray Serve and vLLM fit in. For architects and senior ML engineers.
- Live Course
- 5 modules
-
Balancing p95/p99 latency, throughput and unit cost in model serving: dynamic batching, quantization, caching, GPU/CPU selection and load-testing optimisation. For ML and platform engineers building scalable, cost-effective inference infrastructure.
- Live Course
- 5 modules
-
End-to-end security for inference services: mTLS, OAuth2/JWT, rate limiting, safe serialization, container hardening, defence against model extraction/adversarial attacks and data privacy. For ML, platform and AppSec engineers.
- Live Course
- 5 modules
-
Exporting TensorFlow and Keras models as SavedModel, defining SignatureDefs and serving them over REST and gRPC with TensorFlow Serving. For data scientists and ML engineers starting to deploy TensorFlow models.
- Live Course
- 5 modules
Get in touch
