Course Objective & Scope
Data warehouses built for traditional BI reporting often struggle to deliver the fresh, contextual, traceable and unstructured data required by RAG, agent and machine learning systems. This comprehensive programme shows how to upgrade enterprise data pipelines to an AI-ready standard.
Ingestion from source systems, data quality and data contracts, unified processing of structured and unstructured data, pipelines feeding feature stores and vector databases, lineage and access control are covered hands-on. Participants design an end-to-end data pipeline for one of the organisation's AI use cases and produce an AI-ready data checklist along with a lineage/ownership model.
Who Should Attend?
- Data engineers and analytics engineers supplying data to AI and analytics initiatives
- AI and platform engineers building data infrastructure for RAG and ML systems
- Data architects modernising enterprise data platforms
- Prerequisite: working knowledge of Python, SQL and basic ETL/ELT pipelines
Key Takeaways
- Assess existing data assets against an AI-ready data checklist
- Build unified pipelines producing both tabular features and document embeddings
- Add safeguards against data leakage, schema breakage and data staleness
- Design rich metadata, chunking and incremental indexing flows for RAG and agents
- Embed data ownership, lineage and access-control boundaries into the pipeline
Curriculum
From BI Data to AI-Ready Data
- Data requirements of ML, RAG and agent workloads
- AI-ready data checklist: freshness, context, quality and traceability
Medallion Architecture and Tabular Feature Pipelines
- Cleaning and enriching data across Bronze, Silver and Gold layers
- Point-in-time feature generation and online/offline consistency
Unstructured Data and Embedding Pipelines
- Document parsing, chunking, metadata enrichment and embedding generation
- Incremental upserts and deletes for changing source documents
Data Quality, Data Contracts and Observability
- Producer-consumer contracts, schema validation and drift detection
- Quality gates and quarantining bad records
Lineage, Access Control and Governance
- End-to-end lineage from source document to model response
- PII masking, RBAC/ABAC filtering and the ownership model
Other Courses in This Category
-
Automated data quality testing (dbt, Great Expectations, Soda) and enforceable producer-consumer data contracts to stop silent schema changes and corrupted data from breaking downstream analytics and AI.
- Live Course
- 5 modules
-
Discovering data assets, tracking column-level lineage and catching pipeline delays or distribution drift early using OpenLineage, DataHub/OpenMetadata and data observability practices.
- Live Course
- 5 modules
-
Scalable data pipelines that turn PDFs, Office files, scanned documents, call-centre audio and images into parsed, chunked, metadata-enriched and indexed assets for RAG and multimodal AI.
- Live Course
- 5 modules
-
Lakehouse storage on Apache Iceberg, modular tested transformations with dbt and orchestration with Apache Airflow, for engineers building scalable, open-standard data platforms.
- Live Course
- 5 modules
-
Enterprise data architecture supporting analytics, classical ML, RAG and autonomous agents under one roof: designing lakehouse storage, semantic layers, feature stores, vector indexes and governance.
- Live Course
- 5 modules
-
Writing schema, business-rule, referential integrity and statistical distribution tests with dbt tests, Soda Core and Great Expectations to catch bad data in CI/CD and staging.
- Live Course
- 5 modules
Get in touch
