Course Objective & Scope
Much of an organisation's information lives not in plain text but in PDF pages, invoices, forms, technical diagrams, product photos and call-centre audio. This course covers the multimodal capabilities of GPT-4o, Claude and Gemini, open-weight vision-language models (VLMs) such as Qwen-VL, Pixtral and Llama Vision, and visual/audio token pricing.
Participants compare classic OCR pipelines with direct VLM document understanding and build a structured extraction and visual Q&A application on image, document or audio scenarios from their own organisation.
Who Should Attend?
- Software and AI engineers building document processing, visual inspection or media analysis products
- Data scientists extracting structured information from multimodal data
- Technical teams designing call-centre, field-operations and document automation solutions
- Prerequisite: experience using LLM APIs with Python or TypeScript
Key Takeaways
- Calculate how image resolution, tiling and frame sampling affect token cost and accuracy
- Extract schema-compliant data from complex tables, charts and multi-column PDFs with VLMs
- Process audio and video via transcription + LLM pipelines or native multimodal APIs
- Add verification steps that catch visual hallucinations and reading errors
- Benchmark OCR + LLM against direct VLM pipelines on quality and cost for internal data
Curriculum
How Multimodal Models Work
- Vision encoders, projection layers and unified token spaces
- Image resolution, patch/tile calculation and provider pricing
Visual and Document Understanding (Document AI)
- Structured JSON extraction from scanned documents, forms, tables and charts
- Comparing classic OCR/layout tools (Docling, Surya) with direct VLMs
Open-Weight Vision-Language Models
- Running Qwen-VL, Pixtral and Llama Vision on-premises or in private cloud
- In-house inference for confidential enterprise documents
Audio and Video Processing Workflows
- Whisper/ASR + LLM chains versus native audio-capable models
- Video frame sampling, timestamped event extraction and summarisation
Multimodal Hallucinations, Security and Evaluation
- Visual hallucination modes, small-text OCR errors and crop/zoom techniques
- Indirect prompt injection via images and field-level accuracy evaluation
Hands-On: Multimodal Document and Media Pipeline
- End-to-end extraction service on internal documents or images
- Quality, latency and cost-per-page analysis
Other Courses in This Category
-
Advanced programme on sparse Mixture of Experts (MoE) LLMs, covering gating networks, active vs total parameters, VRAM sizing, expert parallelism in vLLM/SGLang and MoE fine-tuning.
- Live Course
- 6 modules
-
Building low-latency streaming text, partial structured output and bidirectional voice AI applications using SSE, WebSockets and WebRTC, with TTFT optimisation, barge-in handling and concurrency control.
- Live Course
- 6 modules
-
Building fast, low-cost enterprise solutions with 1B–14B Small Language Models (Llama, Qwen, Gemma, Phi, Ministral) for classification, extraction and routing, including distillation and task adaptation.
- Live Course
- 6 modules
-
Deploying and integrating open-weight LLMs (Llama, Qwen, Mistral, DeepSeek) on-premises or in private cloud as OpenAI-compatible APIs using vLLM, SGLang and Ollama, including GPU sizing and quantization.
- Live Course
- 6 modules
-
Engineering techniques to cut token spend and p95 latency in production LLM applications: prompt and semantic caching, Batch APIs, output length control, tiered models and unit economics.
- Live Course
- 6 modules
-
Hands-on programme on building modular LLM applications with modern LangChain (LCEL, ChatModel abstractions, prompt templates, structured outputs, tool calling, retrievers and LangSmith tracing).
- Live Course
- 6 modules
Get in touch
