Model Online / On-Site
Duration
Modules 6 modules

Course Objective & Scope

Two SQL queries that answer the exact same business question can look completely different syntactically (CTEs vs subqueries, different column aliases, reordered projections), making string similarity useless for Text-to-SQL evaluation. Conversely, naive result-table comparison can be fooled by broken queries that happen to return the same rows on a sparse database snapshot.

Drawing on enterprise benchmarks such as BIRD and Spider 2.0, this course covers Execution Accuracy (EX), canonical result-set matching, SQLGlot AST component analysis and query efficiency scoring, guiding participants to build an automated evaluation suite for their own schemas.

Who Should Attend?

  • Data scientists and AI engineers measuring and improving Text-to-SQL accuracy
  • Analytics and QA engineers testing prompt, model and semantic-layer changes
  • Technical leads defining acceptance criteria for conversational analytics products
  • Prerequisite: strong SQL skills and Python experience in data analysis or test automation

Key Takeaways

  • Implement canonical result-set comparators tolerant to column aliases, column ordering and float rounding
  • Prevent false-positive execution matches using targeted database fixtures and edge-case rows
  • Compare JOIN paths, WHERE predicates and aggregations structurally using SQLGlot ASTs
  • Measure query performance alongside correctness with Valid Efficiency Score (VES) and cost metrics
  • Build a stratified enterprise Text-to-SQL benchmark covering simple, multi-join, window, ambiguous and unanswerable queries

Curriculum

Text-to-SQL Evaluation Challenges and Metric Taxonomy
  • Why Exact String Match fails: semantically equivalent SQL formulations and alias/projection variations
  • Execution Accuracy (EX), Canonical Result Match, AST Component Match and Valid Efficiency Score (VES)
Canonical Result-Set Comparison Algorithms
  • Column-order-independent matching and order-sensitive vs multiset row comparison
  • Handling floating-point tolerance, date normalisation, `NULL` semantics and extra informational columns
Eliminating False Positives and AST Structural Scoring
  • Catching accidental matches and empty-set false positives with adversarial database test fixtures
  • Partial scoring of tables, JOIN keys, WHERE predicates and aggregations via SQLGlot AST diffing
Designing an Enterprise Text-to-SQL Benchmark Suite
  • Difficulty tiers: single-table, multi-join, CTE/window functions and domain-rule-heavy questions
  • Evaluating ambiguous questions (multiple valid interpretations) and unanswerable schema-gap questions
Failure Taxonomy and Automated Regression Pipelines
  • Automated error bucketing: schema-linking miss, wrong JOIN path, wrong literal value, dialect error
  • Running deterministic Text-to-SQL regression suites in CI/CD across prompt and model updates
Lab: Building an Automated Text-to-SQL Evaluation Harness
  • Implementing a golden SQL benchmark and canonical result comparator on an analytical schema
  • Comparing model and schema-linking configurations across accuracy tier and latency

Other Courses in This Category

  • RAG with LangChain RAG with LangChain

    Hands-on workshop on building an end-to-end Retrieval-Augmented Generation (RAG) chatbot using LangChain, covering document chunking, embeddings, vector databases, and a Gradio interface.

    RAG & Enterprise Knowledge Systems Corporate, Individual Workshop
    Live Course
    6 modules
    View
  • Multimodal RAG Multimodal RAG

    Hands-on workshop on building a Multimodal RAG system with LangChain to extract, embed, and query text, tables, and images from complex PDF documents.

    RAG & Enterprise Knowledge Systems Corporate, Individual Workshop
    Live Course
    6 modules
    View
  • Retrieval-Augmented Generation (RAG) Fundamentals

    Hands-on fundamentals for building an end-to-end RAG system over enterprise documents: ingestion, chunking, embeddings, vector retrieval, grounded prompting with citations and baseline evaluation.

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View
  • Embeddings and Semantic Search Engineering

    Deep dive into dense, sparse (SPLADE/BM25), Matryoshka and late-interaction (ColBERT) embeddings, Turkish/multilingual retrieval quality, domain adaptation and semantic search engineering.

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View
  • Vector Database Architectures: pgvector, Milvus, FAISS, Weaviate and Pinecone

    Engineering deep dive into ANN indexing (HNSW, IVF, DiskANN, PQ) and vector databases (pgvector, Milvus, Weaviate, FAISS, Pinecone), covering filtered search, multi-tenancy and capacity planning.

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View
  • Data Preparation, Document Parsing and Chunking for RAG

    Parsing complex PDFs, tables and scanned enterprise documents with Docling, Unstructured and VLMs, and applying structural, semantic, parent-child and contextual chunking for high-recall RAG.

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View

Get in touch