Big Data

What is Big Data?

• Big Data Definition and Characteristics (5Vs - Volume, Velocity, Variety, Veracity, Value)

• Which Data Qualifies as "Big Data"?

• In which business processes is it used?

Big Data Lifecycle

• Data Ingestion → Storage → Processing → Analysis → Serving → Governance

Introduction to Relational Databases

• Relational Data Model Fundamentals

• Basic SQL Operations (SELECT, JOIN, GROUP BY, WHERE)

• Normalization Concepts (1NF, 2NF, 3NF) and Importance

• ACID Properties and Importance

Evolution of Data Architectures

• Database → Data Warehouse → Data Lake → Lakehouse Architectures

• Lambda & Kappa Architectures (Context & Importance)

Distributed Systems Fundamentals

• Why distributed systems instead of a single supercomputer?

• CAP Theorem, BASE Concept, and Practical Implications

Setting Up a Cloud Development Environment (AWS/Azure Focused)

• Introduction to Cloud Console & IAM Fundamentals  

• CLI Installation

• Infrastructure as Code (IaC) Fundamentals (Terraform/CloudFormation)

• Docker Fundamentals

Lab 1

• Setting up a simple distributed (HDFS + Spark) and MySQL environment in Docker

• Local "S3" experiment with MinIO

• Creating AWS S3 with Terraform

• Provisioning basic cloud storage with IaC

Traditional (On-Premise) Data Storage Solutions

• On-Premise File Systems (HDFS - Hadoop Distributed File System)

• Hadoop Fundamentals and Ecosystem

• Hadoop Cluster (YARN) Setup and Management

• Hadoop Ecosystem Tools (HDFS, Hive, HBase)

Cloud Fundamentals and Infrastructure Setup

• AWS/Azure/GCP Fundamentals, Console and CLI Usage

• Managing Infrastructure as Code (Terraform, CloudFormation)

Object Storage Services in the Cloud

• AWS S3, Azure Data Lake Storage (ADLS), Google Cloud Storage (GCS)

• Storage Tiers, Pricing, and Lifecycle Management

• Access Management and Security (IAM, Access Control)

Data Formats for Big Data

• CSV and JSON formats and their limitations

• Columnar Storage Formats: Parquet, ORC

• Serialization and Schema Evolution (Avro, Protobuf)

NoSQL Databases

• What is NoSQL? When is it Preferred? (CAP and BASE Connection)

• Key-Value Stores (DynamoDB, Redis, etc.)

  • Use cases, modeling, scaling

• Document Stores (MongoDB, Cosmos DB)

  • Use cases, flexible schema

• Column-Family Stores (Cassandra, HBase)

  • Write-heavy workloads, time series

• Brief Overview of Graph and Time-Series Databases

Lakehouse Architecture

• Data Lake vs Data Warehouse vs Lakehouse

• Delta Lake, Apache Iceberg, Apache Hudi Fundamentals

• ACID, Schema Evolution, Time Travel

Lab 2

• Ingesting and converting data to Parquet in cloud storage

• Basic NoSQL interactions

• Implementing a simple Delta Lake table

Data Ingestion Strategies

• Batch Ingestion vs Streaming Ingestion

• Differences between ETL and ELT

• CDC (Change Data Capture) Approaches (Debezium)

• API-Based Ingestion

Ingestion Tools and Technologies

• Sqoop, Flume

• Apache Kafka and Kafka Connect

• Cloud Tools (AWS Glue, Azure Data Factory, GCP Dataflow)

Data Orchestration

• What is orchestration?

• Why orchestrate?

• Apache Airflow Concepts (DAGs, Operators, Error Handling)

• Cloud Orchestration Tools (AWS Step Functions, Azure Data Factory Pipelines)

• Dependencies, error handling, retries, monitoring

Infrastructure as Code (IaC) for Data Pipelines

• Using Terraform / CloudFormation / ARM Templates

• Managing data resources (Storage, Compute)

• Version control for infrastructure

Lab 3

• Building an ELT pipeline using cloud services (e.g., ADF/Glue)

• Performing data transformations

• Orchestration with Airflow or Step Functions defined via IaC

Spark Architecture & Core Concepts

• Core Concepts (Driver, Executor, Cluster Manager)

• RDD, DataFrame, and Dataset Concepts

• Catalyst Optimizer & Tungsten

• Lazy Evaluation

Spark SQL & DataFrame API

• Reading/writing various sources (Parquet, Delta, JDBC, NoSQL)

• SQL queries & DataFrame operations

  • Select

  • Filter

  • Join

  • GroupBy, Aggregate, Window Functions

• User-Defined Functions (UDFs)

Spark Performance Tuning

• Partitioning, Shuffling, and Join Strategies

• Caching and Persistence Methods

• Using and optimizing via Spark UI

Managing Spark in the Cloud

• Using and Managing AWS EMR, Azure Databricks, GCP Dataproc

• Spark Cluster Tuning and Optimization

Lab 4.1

• Comprehensive hands-on practice with Spark SQL & DataFrame API on real-world data

Lab 4.2

• Deploying, monitoring, and performing basic tuning of Spark jobs on a managed cloud service

Real-Time Data Processing Concepts

• Event Time, Processing Time, Windowing, and Watermarking

• Data Processing Guarantees (Exactly-once, At-least-once)

Streaming Platforms and Kafka

• Apache Kafka Deep Dive (Broker, Topic, Partition, Offset, Consumers)

• Cloud Platforms (AWS Kinesis, Azure Event Hubs, GCP Pub/Sub)

Spark Structured Streaming

• Spark Streaming and Structured Streaming Concepts

• DataFrame API

• Micro-batch vs Continuous Processing Differences

• Managing Streaming Data in the Cloud (Kafka → Spark → Storage)

• Sources & Sinks (Kafka, Files, Cloud Streams, Delta Lake)

• Checkpointing & fault tolerance

Introduction to Apache Flink (Comparative Overview)

• Focus on low-latency streaming

• Key conceptual differences from Spark Streaming

Lab 5.1

• Setting up and interacting with Kafka or a cloud streaming service (Kinesis/Event Hubs/PubSub)

Lab 5.2

• Creating a Spark Structured Streaming job (reading, transforming, windowing, writing)

• Implementing watermarking & checkpointing

Data Quality Management

• Data Quality Dimensions (Accuracy, Consistency, Completeness, etc.)
• Data Quality Tools (Great Expectations, Deequ)

Data Governance Fundamentals

• Metadata Management and Data Catalog (AWS Glue Catalog, Azure Purview)
• Data Lineage Concept and Compliance (GDPR, CCPA)

Big Data Security

• Authentication and Authorization (IAM, RBAC)
• Data Privacy Techniques (Anonymization, Pseudonymization)
• Encryption (At Rest and In Transit)

Big Data Analytics Tools

• Distributed SQL Engines (Presto, Trino, AWS Athena, Google BigQuery)

• Cloud Data Warehouses (AWS Redshift, Azure Synapse, Google BigQuery)

Modern Cloud Data Warehouse

• AWS Redshift / Azure Synapse Dedicated SQL / Google BigQuery

• MPP architecture, columnar storage, distribution, WLM

• Loading from a Lakehouse

• When to use a data warehouse vs. querying the lake directly

Integration with Business Intelligence (BI) Tools

• Connecting Tableau, Power BI, Looker, and AWS QuickSight

• DirectQuery and Data Import Strategies

• Query engine endpoints (Athena, Synapse, BigQuery, Databricks SQL)

Introduction to Data APIs and Serving Layers

• Using serverless functions (Lambda/Azure Functions) for simple data serving

Lab 6

• Querying Lakehouse data using Athena/Synapse Serverless/BigQuery

• Connecting a BI tool

• Building a simple serverless data API

Monitoring, Logging, and Alerting

• Distributed monitoring concepts

• Key metrics for pipelines (Spark, Kafka, Storage, Query Engines)

• Cloud Monitoring Tools (CloudWatch, Azure Monitor, Google Monitoring)

• Logging and Alerting in Distributed Systems

Testing Data Pipelines

• Unit testing (Spark)

• Integration testing stages

• Data validation testing (Data quality integration)

• Test data strategies & canary deployments

CI/CD Approaches

• Continuous Integration and Delivery with GitHub Actions, GitLab CI, Jenkins, and Azure DevOps

• Testing Strategies and Environment Management (Development, Stage, Production)

Cost Management and Optimization

• Cloud Services Pricing Models

• Resource Optimization (Reserved Instances, Spot Instances)

• Storage tier optimization

• Cost monitoring tools & budgeting

• Query cost optimization

Lab 7

• Instrumenting jobs with metrics and logs

• Creating alerts

• Building a basic CI/CD pipeline

• Analyzing and recommending cost optimizations for a sample workload

Emerging Trends

• Data Mesh Concept and Applications

• MLOps and Feature Stores

• Vector Databases and the AI Connection

Career and Continuous Learning

• Data Engineering Career Paths

• Continuous Learning Strategies and Resources

End-to-End Big Data Project

• Data Ingestion, Storage, Processing, Analytics, and Visualization with a Real-World Scenario

• Integration of IaC, CI/CD, Security, Monitoring, and Cost Optimization

• Project Documentation and Demo/Presentation