Engineering at Trillion-Row Scale: A Deep Dive into Uber’s Hudi-Powered Data Lake

In the world of data engineering, “scale” is often a relative term. But at Uber, scale means managing a multi-hundred-petabyte repository that handles 6 trillion rows ingested daily. To manage this tidal wave of information, Uber moved away from traditional append-only data lakes to create Apache Hudi™, a storage engine that brings database-like primitives to the lake.

This post explores the technical architecture and workflows that allow Uber to maintain data freshness and consistency at one of the largest centralized data lakes on the planet.

——————————————————————————–

1. The Numbers Behind the Machine

To appreciate the engineering, you first have to understand the sheer magnitude of Hudi’s footprint at Uber:

• 19,500 Hudi datasets across domains like mobility, delivery, and safety.
• 350 logical petabytes stored on HDFS™ and Google Cloud Storage.
• 10 petabytes ingested daily, with 3 petabytes written to the lake every 24 hours.
• 350,000 commits and 70,000 table service operations (compaction, cleaning, clustering) daily.
• 4 million analytical queries served weekly via Presto® and Apache Spark™.

——————————————————————————–

2. The Complete Technical Workflow

Uber’s data platform is a deeply interconnected stack where Hudi acts as the foundational storage orchestrator.

Step 1: Ingestion (Spark & Flink)

• Spark Batch Ingestion: Used for large-scale hourly/daily pipelines and backfills. Uber uses an internal scheduler called Piper to manage these DAGs and enforce SLAs.
• Flink Streaming Ingestion: To achieve sub-15-minute freshness, Uber is shifting to Flink-native streaming. This allows for continuous, low-latency writes that support real-time business lines.

Step 2: Storage & Indexing (The Hudi Magic)

As data enters the lake, Hudi uses specialized primitives to manage it:

• ACID Transactions: Ensures correctness even if pipelines overlap or fail.
• Record Index (RI): This is a native, HFile-backed data structure that maps record keys to file groups. It enables O(1) lookups with a latency of just 1-2 milliseconds, even in tables exceeding 300 billion rows.
• Metadata Table (MDT): Instead of overwhelming the HDFS NameNode with file listings, Hudi uses the MDT (an SSTable-based key-value store) to track file locations and column statistics, making folder listings an O(1) operation.

Step 3: Table Services

To keep queries fast, Hudi runs asynchronous Table Services:

• Compaction: Merges small files and updates.
• Clustering: Reorganizes data layout for better query performance.
• Cleaning: Removes older versions of data to save space.

Step 4: Querying & Analytics

• Apache Spark: Used for complex ETL, feature engineering, and ML workloads.
• Presto: The go-to engine for high-concurrency, interactive analytics.
• Incremental Processing: One of Hudi’s biggest advantages is allowing engines to read only what changed since the last commit, avoiding expensive full-table scans.