Apache Hudi

Walmart ran Apache Hudi across 600,000+ Spark cores. They picked it via weighted scorecard against Iceberg and Delta. Scored on availability, compatibility, cost, performance, roadmap, support, and TCO across two workloads: • WL1 — time-partitioned batch with late-arriving records, >99.9% inserts • WL2 — bounded unpartitioned streaming, >99.999% updates Results: • WL1 ingestion: Hudi achieved 5× speedup over legacy ORC • WL2 upsert: "Hudi MoR with sync or background compaction was the only open file format able to handle this pattern" Why Hudi: • Schema evolution • Only one supporting Spark 2.4.x — given Walmart's 600K+ Spark/Hadoop cores across internal cloud + GCP + Azure • Compatible across Hive, Spark, Presto/Trino, BigQuery, Flink They called the migration "a major transformational shift." At Fortune 1 scale, format choice is a multi-year, multi-cloud, multi-engine commitment. #ApacheHudi #DataEngineering #Walmart

Semantic search is becoming a core requirement for AI applications, but keeping embeddings, metadata, updates, deletes, and filters consistent across separate systems adds unnecessary complexity. In our latest blog, we introduce native vector search in Apache Hudi, including the new VECTOR type, hudi_vector_search, and hudi_vector_search_batch. The post walks through how Hudi brings embeddings closer to the lakehouse table so teams can combine semantic search with structured filters, incremental updates, time travel, and SQL-based workflows for RAG, recommendations, document intelligence, and similarity search.

The Hudi file group is the storage abstraction that makes AI-native file formats, RFC-80 column families, and wide AI tables all click into place. A file group is a self-contained set of files that Hudi indexes, versions, and concurrency-controls. One record key maps to one file group. The timeline, metadata table, indexes, and writers all hang off this primitive. Two AI workloads land on it naturally: 🧩 Lance fragments map 1:1. Switching the base file format to Lance keeps file-group semantics intact — same upsert, same indexes, same time travel — but each file group is now a Lance fragment with IVF-PQ vector indexing built in. 🗂️ RFC-80 column families split the file group along column boundaries. A 1,000-column ML feature table fans out into per-family base + log files. Read a subset of features, scan only those files. Update a subset, touch only those files. Iceberg and Delta treat "the file" as the unit. Hudi treats the file group — a logical unit with a lifetime, an identity, a place in the timeline. That abstraction is what lets one design absorb Parquet, Lance, and column-family layouts without rewriting the table mechanics. For AI/ML workloads where physical layout is the optimization, that's the point. #ApacheHudi #AINativeLakehouse #LanceFormat

The small file problem on S3 isn't a curiosity. It's how queries slow down, pipelines stall, and storage bills creep. Three failure modes from too many small files: 🦥 Slow queries — every file is at least one S3 request; rate-limits trigger; fixed per-request overhead dominates 🏗️ Slow pipelines — more files = more tasks = more scheduling overhead; memory pressure on driver 📦 Storage inefficiencies — lower compression ratios, more metadata overhead per file Hudi auto-sizes files at write time. Default Parquet target: 120 MB (hoodie.parquet.max.file.size). Small-file threshold: 100 MB (hoodie.parquet.small.file.limit). Hudi bin-packs new records into existing small files until they hit the target. Exception: bulk_insert skips auto-sizing for raw throughput. Pair it with periodic clustering to consolidate. Applied Intuition runs on the file sizing service — 20× compression vs raw Parquet by sizing to \~512 MB targets. #ApacheHudi #DataEngineering #CloudStorage

Hudi Streamer (formerly DeltaStreamer) is the ingestion utility that does the boring stuff for you. Out of the box: 📥 Exactly-once ingestion from Kafka, DFS folders, Sqoop incremental imports, HiveIncrementalPuller 📄 JSON, Avro, or custom record types 🔁 Checkpoint management, rollback, recovery 🧩 Schema fetched from DFS or Confluent Schema Registry 🛠️ Pluggable transformations 🗂️ Multi-table mode (HoodieMultiTableStreamer) — one job, N Kafka topics, N Hudi tables Continuous mode for streaming jobs. Once-and-done mode for batch backfills. Same binary, different --source-class and --continuous flag. Funding Circle uses MultiTableStreamer for 80 topics today, growing to 200+. The Project Kirby framework wraps it with YAML pipeline definitions that engineers commit to their own repos. #ApacheHudi #DataEngineering #Kafka

Notion saved $1M+ in 2022 alone by moving their data lake from Snowflake-via-Fivetran to Apache Hudi. The pain that forced the rewrite: 📈 Block data doubled every 6-12 months (20B rows in 2021 → 200B by 2024) 🔌 480 Fivetran connectors created unsustainable on-call burden 🔄 90% of Notion's upserts were updates — warehouses optimize for inserts 🌳 Permission hierarchies needed expensive SQL tree traversal, impossible at scale What they built: • Debezium CDC connectors → Kafka (capture Postgres changes) • Apache Hudi DeltaStreamer → S3 (update-heavy workload) • Apache Spark → complex transformations Why Hudi over Iceberg or Delta in 2022: strong update-heavy performance + open-source + native integration with Debezium CDC messages. Iceberg + Delta weren't optimized for update-heavy then. Results: • $1M+ savings in year one • Ingestion: >1 day → minutes-to-hours • Powered Notion AI and Search The lesson: when your workload is 90% updates, picking a format designed for it changes the cost math. #ApacheHudi #DataEngineering #Notion

You don't need Spark to read a Hudi table. hudi-rs is the native Rust implementation with Python and C++ bindings — built for the AI/ML toolchains that already live outside the JVM. What hudi-rs ships: 📸 Snapshot, time-travel, and incremental queries with filtering 🌊 Streaming reads — memory-efficient batch processing without loading full results 🗂️ Direct file group access (experimental) for finer-grained workflows ☁️ Native cloud storage support — S3, Azure, GCS with credential management 🧩 DataFusion extension — query Hudi tables from DataFusion-based APIs 🔗 Integrations with Daft and Ray Why this matters for AI workloads: the dominant AI/ML toolchains are Python (PyTorch, transformers, scikit-learn) and increasingly Rust (vector DBs, fast embedding generation). Forcing every Hudi access through the JVM is friction that drives teams toward less efficient alternatives. hudi-rs eliminates that friction. Same Hudi tables, queryable from notebooks, ML inference servers, Python DataFrames, Ray actors, DataFusion-backed APIs — without a Spark cluster in the path. Install: `pip install hudi` for Python, the `hudi` crate via Crates.io for Rust. #ApacheHudi #Python #Rust #AIEngineering

JD.com built a 125+ petabyte retail data lake on Apache Hudi. Record-level mutations on commerce data at planetary scale. The workload profile: • E-commerce retail data with constant updates (orders, inventory, pricing, returns) • Multiple business lines requiring consistent reads across the same datasets • Latency requirements that batch ETL couldn't meet The choice: Hudi for record-level mutations + incremental queries. Specific scenarios across recommendation, supply chain, and operational analytics saw substantial latency improvements and cost reductions. JD.com hosted the 2025 Apache Hudi Asia Meetup. The level of community investment matched the production scale. For commerce engineering teams comparing lakehouse formats: 125+ PB at JD.com is a real existence proof. #ApacheHudi #DataEngineering #ECommerce #JDcom

Automated schema evolution on Hudi + AWS Glue. The "upstream added a column" problem, solved. The architecture: Aurora → AWS DMS (CDC) → S3 raw bucket (Parquet). EventBridge → Lambda reads table configs from DynamoDB → triggers Glue jobs per table in parallel → Hudi ingestion with native Glue Hudi support → Hive sync auto-registers tables in AWS Glue Data Catalog → Athena queries. Handled automatically: new columns appear downstream immediately; renamed columns — old preserved (empty), new populated; deleted columns — empty retained for backward compatibility. The DynamoDB config table is the operating surface: one row per Hudi table (Glue job, status, partition/record keys, schema). When this fits: many source tables evolving independently, schema changes that propagate without manual ALTER TABLEs, a file-based lake over a warehouse for cost, petabyte-scale automation. CloudFormation provisions Lambda + DynamoDB + S3 + Glue jobs. Drop in your configs. #ApacheHudi #DataEngineering #AWS #SchemaEvolution

Out-of-line BLOBs in Hudi — the answer to "we have terabytes of video, can the lakehouse govern it without storing every byte in the row?" The table row stores a pointer triple: external_path + offset + length. The actual bytes live in the native binary file format or in a binary container file on S3 / cloud storage. Table footprint stays at <1% of total data size. The container pattern matters in many cases. Packing many objects into one container file avoids millions of small S3 objects; the BLOB row tells you exactly where each object's bytes start and end. It's also flexible to refer to different parts of the same binary file from multiple rows e.g video file with start frames. Reading: 📌 Standard projection on a BLOB column returns the descriptor — fast, metadata only 🧾 read_blob(content) materializes the bytes on demand from the external path The discipline: filter first, then resolve. SELECT read_blob(content) FROM large_table without a WHERE clause fetches everything. Inline vs out-of-line is a per-row decision, not a per-table one. Small images inline, large videos out-of-line, in the same Hudi table. #ApacheHudi #UnstructuredData #AINativeLakehouse