Introduction
Modern data engineering is more than just moving data from one place to another. A production-ready pipeline must be scalable, reliable, secure, cost-efficient, and capable of handling failures gracefully.
To explore these concepts, I built an end-to-end serverless Data Engineering pipeline on AWS that ingests live YouTube Trending data through the YouTube Data API v3, processes it using the Medallion Architecture (Bronze → Silver → Gold), validates data quality, and delivers analytics-ready datasets for business reporting.
This project demonstrates how multiple AWS services work together to build an automated, production-ready data platform.
🏗 Solution Architecture
The pipeline is built using a fully serverless architecture where each AWS service performs a dedicated responsibility.
Data Flow
YouTube Data API
│
▼
Amazon EventBridge (Scheduler)
│
▼
AWS Lambda (Data Ingestion)
│
▼
Amazon S3 Bronze Layer
│
▼
AWS Step Functions
│
┌──────┴─────────┐
│ │
▼ ▼
AWS Lambda AWS Glue
Reference ETL Bronze → Silver ETL
│ │
└──────┬─────────┘
▼
Data Quality Validation
│
▼
AWS Glue (Silver → Gold)
│
▼
Amazon Athena & QuickSight🥉 Bronze Layer – Raw Data Ingestion
The Bronze layer stores raw API responses exactly as received from YouTube.
Key Features
- Automated ingestion using Amazon EventBridge
- Live data from YouTube Data API v3
- Raw JSON stored in Amazon S3
- Metadata captured during ingestion
- Region-based partitioning
- Immutable raw storage for auditing and replay
Keeping the raw data untouched ensures complete traceability and enables future reprocessing if business logic changes.
🥈 Silver Layer – Data Cleansing & Standardization
The Silver layer transforms raw JSON into clean, analytics-ready datasets.
Processing Includes
- Schema enforcement
- Data type casting
- Null handling
- Deduplication
- Date standardization
- Region normalization
- Derived metrics
- Parquet conversion
- Glue Catalog registration
The pipeline processes statistics data and reference category data in parallel to reduce execution time.
✅ Data Quality Framework
Before promoting data to the Gold layer, a dedicated validation stage verifies the quality of processed datasets.
Validation Checks
- Row count validation
- Schema validation
- Null value checks
- Freshness validation
- Missing column detection
- Numeric range validation
- Business rule verification
Only datasets passing every validation proceed to downstream analytics.
🥇 Gold Layer – Business Analytics
The Gold layer contains curated datasets optimized for reporting and analytics.
Business Tables
- Trending Analytics
- Channel Analytics
- Category Analytics
These datasets are optimized for:
- Amazon Athena
- Amazon QuickSight
- SQL Analytics
- BI Dashboards
⚙ Workflow Orchestration with AWS Step Functions
The entire pipeline is orchestrated using AWS Step Functions, providing centralized control over execution.
Workflow Features
- Sequential orchestration
- Parallel processing
- Wait states
- Choice states
- Retry mechanisms
- Error handling
- Success and failure notifications
This eliminates manual intervention and enables reliable end-to-end automation.
🔄 Idempotent & Incremental Processing
A production pipeline must safely support reprocessing.
This project implements:
- Partition overwrite strategy
- Glue Job Bookmarks
- Duplicate removal
- Incremental processing
- Safe re-execution without duplicate records
These practices ensure data consistency even when workflows are retriggered.
🔐 Security & Governance
Security and governance were built into the solution from the beginning.
Implemented Features
- IAM Least-Privilege Access
- AWS Glue Data Catalog
- Metadata Management
- Schema Evolution
- Dataset Registration
- Secure service-to-service communication
📈 Monitoring & Alerting
Operational visibility is essential for production pipelines.
The solution includes:
- Amazon CloudWatch Logs
- Structured logging
- AWS Step Functions execution history
- Amazon SNS success notifications
- Amazon SNS failure alerts
Failures in ingestion, transformation, or data quality are immediately reported.
💰 Cost Optimization
The pipeline follows AWS cost optimization best practices.
Optimizations
- Fully serverless architecture
- Pay-per-use services
- Partitioned Parquet storage
- Snappy compression
- Predicate pushdown
- Incremental processing
- S3 Lifecycle policies
- Automatic archival of historical Bronze data to Amazon S3 Glacier
These optimizations reduce storage costs while improving query performance.
🛠 Tech Stack
Cloud Services
- AWS Lambda
- AWS Glue
- AWS Step Functions
- Amazon S3
- Amazon Athena
- Amazon QuickSight
- Amazon EventBridge
- Amazon SNS
- Amazon CloudWatch
- AWS Glue Data Catalog
- AWS IAM
Programming
- Python
- PySpark
- SQL
Data Formats
- JSON
- Parquet