Course Objective & Scope
In multimodal systems failures rarely stem from text generation alone: a single misread digit on a blurry invoice, a dropped negation in an audio stream or a missed video frame can corrupt the entire result. This course teaches how to evaluate vision, document, audio and video pipelines at both the component and end-to-end level.
Participants implement document extraction metrics (ANLS, TEDS, field F1), spatial grounding IoU, voice quality and latency metrics (WER, WERR, MOS, TTFB) and VLM-as-a-judge graders, delivering an automated eval suite and measurement plan for their organisation's multimodal application.
Who Should Attend?
- AI, computer vision and speech engineers building multimodal systems
- LLMOps, quality assurance (QA) and evaluation infrastructure teams
- Technical product managers and data scientists
- Prerequisite: Python and basic familiarity with model/LLM evaluation
Key Takeaways
- Separate perception errors (OCR/STT/vision) from downstream reasoning errors in your metrics
- Compute field-level F1, table structure TEDS and bounding-box IoU automatically for documents and screens
- Measure entity recall (WERR), barge-in failure rates and stage-level latency percentiles in voice agents
- Calibrate VLM and Audio-LLM judges against human annotations while controlling judge biases
- Run synthetic perturbation tests (noise, compression, rotation, blur) to verify pipeline robustness
Curriculum
Multimodal Error Taxonomy: Perception vs Reasoning
- Isolating root causes: media capture quality, encoder/OCR/STT or LLM reasoning
- Component-level versus end-to-end evaluation architectures
Vision, Screen and Document Evaluation Metrics
- Structured field F1, ANLS, table TEDS and spatial IoU/pointing accuracy
- Testing for visual hallucinations (fabricating non-existent objects or text)
Voice and Real-Time Dialogue Evaluation
- Beyond raw WER: keyterm/entity error rates and semantic preservation
- MOS/UTMOS naturalness, turn-taking latency (p50/p95) and false barge-in rates
VLM-as-a-Judge and Synthetic Perturbation Testing
- Rubric-based multimodal judge prompts and human correlation calibration
- Injecting blur, JPEG compression, skew and acoustic background noise systematically
Automated Eval Pipelines and Measurement Plan
- Versioning media datasets and running cost/latency/accuracy Pareto analyses
- Building a CI/CD multimodal eval suite for an enterprise use case
Other Courses in This Category
-
Comprehensive introduction to multimodal AI models that understand and generate text, images, audio and video: model families, enterprise use cases, capability limits, cost/latency trade-offs and model selection.
- Live Course
- 5 modules
-
End-to-end design of enterprise applications combining text, image, audio and document inputs: media preprocessing, structured outputs, hybrid model routing and cost/latency optimisation.
- Live Course
- 5 modules
-
High-accuracy extraction from complex PDFs, invoices, contracts, tables and forms using VLMs and modern document parsers: layout understanding, schema validation, bounding-box grounding and human-in-the-loop review.
- Live Course
- 5 modules
-
Multimodal RAG architecture over enterprise documents containing tables, charts, diagrams and images: visual-text indexing, ColPali and image embeddings, hybrid retrieval and visually grounded answer generation.
- Live Course
- 5 modules
-
Conversation design for LLM-based chat and voice assistants: persona and tone guidelines, multi-turn dialogue flows, disambiguation, repair strategies and seamless handoff to human agents.
- Live Course
- 5 modules
-
Designing AI customer service agents and human-agent assist systems powered by knowledge bases (RAG), CRM/order tools and policy guardrails, from contact centre architecture to production rollout.
- Live Course
- 5 modules
Get in touch
