Course Objective & Scope
General-purpose multimodal APIs often fall short on domain-specific imagery—such as industrial quality control, engineering schematics or medical scans—or cannot be used due to strict on-premise data residency rules. This advanced engineering course covers the internal architecture and domain adaptation of multimodal models.
Participants examine vision and audio encoders (SigLIP, ViT, Whisper), projection/adapter layers and dynamic high-resolution tiling, fine-tune an open-weight model (such as Qwen-VL, Pixtral or Gemma) with LoRA/QLoRA on domain data, and serve it efficiently with vLLM.
Who Should Attend?
- AI and deep learning engineers training and adapting models
- Computer vision and speech processing researchers
- MLOps teams deploying self-hosted multimodal model infrastructure
- Prerequisite: PyTorch, transformer architecture fundamentals and basic fine-tuning experience
Key Takeaways
- Compare modular encoder–projector–LLM architectures against native early-fusion multimodal transformers
- Optimise visual token counts, dynamic resolution tiling and GPU memory footprints
- Curate high-quality visual instruction-tuning datasets from domain image–text pairs
- Fine-tune an open-weight VLM with LoRA/QLoRA while preventing catastrophic forgetting
- Quantise and serve the adapted multimodal model at low latency using vLLM or SGLang
Curriculum
Multimodal Architectural Patterns and Representation Spaces
- Contrastive encoders (CLIP/SigLIP), MLP/Q-Former projectors and cross-attention
- Late-fusion adapters versus native early-fusion transformers
Visual and Audio Tokenisation
- Patch embeddings, dynamic high-resolution tiling (AnyRes) and video token compression
- Continuous audio features versus discrete neural audio codec tokens
Multimodal Dataset Curation and Synthetic Data
- Formatting visual QA, bounding-box grounding and structured JSON training pairs
- Data filtering, resolution balancing and negative samples to curb hallucination
LoRA and QLoRA Fine-Tuning on Open-Weight VLMs
- Deciding when to freeze or unfreeze the vision encoder and projector
- Hands-on fine-tuning with Qwen-VL, Llama Vision or Pixtral
Quantisation and High-Throughput Multimodal Serving
- Impact of AWQ/FP8 quantisation on fine-grained visual accuracy
- Encoder caching and low-latency inference with vLLM and SGLang
Other Courses in This Category
-
Building visual inspection and analysis applications with Vision-Language Models (VLMs) in manufacturing, retail, insurance and field operations: visual grounding, multi-image comparison and hybrid CV+VLM pipelines.
- Live Course
- 5 modules
-
End-to-end Speech-to-Speech (S2S) models and autonomous voice agents: acoustic emotion and prosody perception, voice tool calling, multi-agent call handoffs and production architecture with Pipecat and LiveKit.
- Live Course
- 5 modules
-
Streaming infrastructure engineering for real-time AI applications: WebRTC, WebSockets and SSE protocols, OpenAI Realtime and Gemini Live API integration, audio/video buffer management and horizontal scaling.
- Live Course
- 5 modules
-
Human-like turn-taking and millisecond latency engineering in voice AI: acoustic VAD, semantic endpointing, false-interruption suppression, TTFT/TTFB optimisation and speculative execution.
- Live Course
- 5 modules
-
Parsing dense documents and application screens (GUIs) at the pixel, coordinate and structural-tree level: OmniParser, Qwen-VL, Docling, UI widget detection and unified document/screen understanding models.
- Live Course
- 5 modules
-
Evaluation engineering for multimodal systems across vision, documents, audio and video: OCR/extraction metrics, grounding IoU, speech WER/MOS and turn-taking metrics, VLM-as-a-judge and regression pipelines.
- Live Course
- 5 modules
Get in touch
