Course Objective & Scope
Product catalogues, technical drawings, field photos and reports carry information in both pixels and text; processing them with separate models loses context. Starting from shared embedding spaces built with contrastive models such as CLIP and SigLIP, this course covers the architecture and practical use of Qwen2.5-VL, LLaVA, InternVL and cloud multimodal LLMs (GPT-4o, Claude, Gemini).
Zero-shot classification, cross-modal search, structured JSON extraction, visual RAG with ColPali and QLoRA fine-tuning of open-weight VLMs on company data are practised hands-on. Participants build a working multimodal search or visual assistant prototype on their own data.
Who Should Attend?
- AI engineers building products that combine computer vision and natural language processing
- Data scientists working on e-commerce, catalogue management, media archives and technical documentation
- LLM developers adding visual understanding to enterprise RAG and agent systems
- Prerequisite: PyTorch, transformer fundamentals and experience with LLM APIs or training
Key Takeaways
- Build text-to-image and image-to-image semantic search with CLIP and SigLIP
- Serve open-weight VLMs (Qwen2.5-VL, InternVL) at high throughput on vLLM
- Design multimodal RAG over visually rich PDFs, charts and diagrams using ColPali and VLMs
- Prepare an instruction dataset and fine-tune a VLM with QLoRA for domain-specific tasks
- Detect visual hallucinations and grounding errors and build an evaluation benchmark
Curriculum
Contrastive Multimodal Representations: CLIP and SigLIP
- Aligning image and text encoders in a shared embedding space
- Zero-shot classification, cross-modal retrieval and vector database integration
Generative Vision-Language Architectures
- Vision encoder, multimodal projector (MLP, Q-Former) and LLM backbone
- Dynamic resolution and visual grounding in Qwen2.5-VL, InternVL and LLaVA-OneVision
Structured Extraction and Multimodal RAG
- Generating schema-bound JSON from images, charts and screenshots
- RAG over complex PDFs and catalogues with ColPali and visual embeddings
VLM Fine-Tuning and Domain Adaptation
- Preparing visual instruction-tuning datasets and synthetic QA generation
- Training vision projectors and LLM layers with TRL and PEFT (QLoRA)
Evaluation, Hallucinations and Serving
- Testing for visual hallucinations, OCR errors and spatial reasoning
- Multimodal batching, image-token budgets and caching with vLLM
Other Courses in This Category
-
Surface defect detection on production lines, unsupervised anomaly detection with anomalib (PatchCore, PaDiM, EfficientAd), industrial camera/lighting design and PLC integration, for quality and manufacturing engineering teams.
- Live Course
- 5 modules
-
Inpatient fall prevention, ICU monitoring, hand-hygiene and PPE compliance, operating-room instrument counting and privacy-preserving edge anonymisation, for hospital digital transformation and healthtech teams.
- Live Course
- 5 modules
-
DICOM/NIfTI processing, 2D/3D lesion and organ segmentation on CT/MRI/X-ray with MONAI and nnU-Net, whole-slide pathology (WSI) analysis and clinical validation, for medical imaging and biomedical AI teams.
- Live Course
- 5 modules
-
Reception and lobby queue/crowd analytics, buffet tray replenishment and food-waste measurement, car-park ANPR, pool/facility safety and privacy-compliant anonymous analytics, for hospitality technology teams.
- Live Course
- 5 modules
-
Model conversion with ONNX Runtime, TensorRT, OpenVINO and TFLite, INT8 quantization, camera pipelines on NVIDIA Jetson and Raspberry Pi, GStreamer and OTA updates, for teams running vision models at low latency in the field.
- Live Course
- 5 modules
-
Knowledge distillation, structured pruning, Quantization-Aware Training (QAT), NAS and Torch-TensorRT profiling, for engineers shrinking and accelerating large computer vision models with minimal accuracy loss.
- Live Course
- 5 modules
Get in touch
