Open Table Formats (Iceberg, Delta Lake, Hudi)¶

Scope¶

Open table formats add database semantics -- ACID transactions, snapshot isolation, schema evolution, time travel, and row-level mutation -- to files sitting in object storage. This file covers Apache Iceberg, Delta Lake, and Apache Hudi at the level of detail needed to choose one and operate it: metadata layer and manifest structure, hidden vs explicit partitioning, partition and schema evolution, copy-on-write vs merge-on-read, snapshot expiry and orphan-file cleanup, compaction and the small-file problem, catalog requirements and commit atomicity, the engine support matrix, and the recurring operational cost of each.

general/data-analytics.md already carries the ADR: Open Table Format Selection decision record with the high-level drivers. This file is the depth behind that ADR -- read it when the decision is live, when an existing table format is misbehaving, or when a second engine needs to read tables a first engine wrote. For the compute layer that reads these tables see general/query-engines.md; for the layering pattern built on top see patterns/lakehouse-medallion.md.

Format-level facts below were checked against the project specifications and source in mid-2026. This area moves fast: verify version-dependent claims (spec versions, engine DML support, default property values) against current documentation before making a commitment on them.

Checklist¶

Format Selection and Commitment¶

[Critical] Has the organization accepted that table-format choice is a multi-year commitment rather than a per-table decision? The format determines which engines can write (reading is far more portable than writing), which catalog is viable, which maintenance jobs must run forever, and which vendors can be dropped later. Splitting a lake across two formats doubles the maintenance surface and usually produces a set of tables only one engine can safely mutate.
[Critical] Is the choice being driven by the write side rather than the read side? Nearly every engine can read all three formats to some degree; the constraint that actually binds is which engines can perform MERGE/UPDATE/DELETE and run maintenance. Enumerate the writers first (Spark, Flink, the warehouse, an ingestion SaaS) and check DML support for each before selecting.
[Critical] Is Iceberg the default unless there is a specific reason to choose otherwise? As of 2026 Iceberg has the broadest independent-engine write support, an open REST catalog specification, and adoption by Snowflake, AWS (S3 Tables), Google, Confluent, and Databricks. That is a statement about optionality, not about technical superiority -- Delta is better integrated where Databricks is the platform, and Hudi is genuinely better at high-frequency upserts. Choose deliberately, but treat a non-Iceberg choice as needing a written justification.
[Critical] If the platform is Databricks-centric, is Delta Lake chosen for managed tables rather than forcing Iceberg onto a Databricks-shaped platform? Databricks now supports Unity Catalog managed Iceberg tables with an Iceberg REST catalog endpoint, but Delta remains the path with the fewest feature gaps inside Databricks. See providers/databricks/data-platform.md.
[Recommended] Is Hudi selected only when the workload is genuinely upsert-dominated -- CDC replication of mutable operational tables, or streaming ingestion with frequent late-arriving updates to old partitions? Hudi's record-level indexing exists to avoid joining incoming changes against the whole table to locate the files to rewrite. If the workload is append-mostly with occasional restatement, that machinery is operational cost without payoff.
[Recommended] Has the Iceberg spec version been chosen explicitly rather than accepted by default? Spec v1, v2, and v3 are all complete and adopted; v4 is under active development and not adopted. The Iceberg library still defaults format-version to 2. v3 adds deletion vectors, mandatory row lineage, variant/geometry/geography types, nanosecond timestamps, and default values -- but engine support for v3 lags well behind v2. Pick v2 for maximum interoperability today; pick v3 only after confirming every reader and writer in scope supports it.
[Optional] Has translation tooling (Delta UniForm, Apache XTable) been evaluated honestly rather than assumed to solve interoperability? UniForm is one-way: Iceberg and Hudi clients can read the generated metadata, but writing to a UniForm table from an external engine can corrupt the underlying Delta table. Apache XTable is still incubating with a long release gap. Translation is a read-compatibility bridge, not a way to run two formats as one.

Metadata Layer and Catalog¶

[Critical] Is it understood that all three formats keep their metadata as files in object storage, and that the catalog exists to hold one mutable pointer atomically? Iceberg's chain is metadata.json -> manifest list (one per snapshot) -> manifest files -> data/delete files; the catalog holds the pointer to the current metadata.json and must swap it via compare-and-set. The Iceberg spec deliberately does not standardize how that swap happens -- it is the catalog's job -- which is exactly why catalog choice is a correctness decision, not a convenience one.
[Critical] Is a catalog with a real atomic-commit mechanism in use, and specifically not the Iceberg Hadoop/filesystem catalog on S3? The filesystem scheme relies on atomic rename; Iceberg's own documentation states it is unsafe on object stores and local filesystems, and the scheme is deprecated for removal in spec v4. Concurrent writers against a Hadoop catalog on S3 can lose commits silently.
[Critical] For Delta Lake on S3 with multiple writers from different clusters, is the DynamoDB-backed log store (delta-storage-s3-dynamodb) configured, or are writes funnelled through a single coordinator or a catalog-managed table? Open-source Delta's default S3 log store is single-driver; the documentation warns that concurrent writes to the same table from multiple Spark drivers can lose data. Despite S3 gaining conditional writes, OSS Delta's answer is still the external log store or catalog-managed commits, not If-None-Match.
[Critical] Does the catalog choice match the governance requirement, given that access control lives in the catalog and not in the format? Options in practice: Iceberg REST catalog (open specification, the portable choice), Apache Polaris (an Apache top-level project since early 2026, REST-native, supports multi-table transactions on local catalogs), AWS Glue Data Catalog (native to AWS, also exposes an Iceberg REST endpoint), Databricks Unity Catalog, Snowflake Horizon, Project Nessie (git-style branches and tags), AWS S3 Tables (managed table buckets with built-in maintenance), and Hive Metastore (legacy, still widespread).
[Recommended] If multi-table atomic commits are a requirement, has it been verified against the specific catalog rather than assumed? The Iceberg REST specification defines POST /v1/{prefix}/transactions/commit for atomic multi-table updates, and Polaris implements it for local catalogs. The Glue Iceberg REST endpoint does not list it. Nessie provides atomic multi-table visibility via branch merge -- Iceberg's own Nessie documentation is explicit that this is a fast-forward merge of separate commits, not a true multi-table database transaction.
[Recommended] Is the catalog's metadata-size ceiling known and monitored? AWS Glue Data Catalog supports Iceberg tables with metadata up to 50 MB (5 MB for federated catalogs) and rejects requests to tables exceeding it. An unmaintained high-commit-rate table does not merely get slow -- it becomes unreadable. This is the sharpest available argument for scheduling maintenance from day one.
[Recommended] Are Iceberg branches and tags used for write-audit-publish or reproducible reporting where appropriate, with their semantics understood? Querying a tag uses that snapshot's schema, while querying a branch uses the table's current schema -- a real source of surprise. On AWS S3 Tables, creating any user-defined branch or tag disables automated snapshot management for the whole table.
[Optional] For Delta, is the transaction-log configuration understood -- 20-digit zero-padded JSON commits in _delta_log/, a checkpoint every 10 commits by default (delta.checkpointInterval), and V2 checkpoints with sidecar files for very large tables? Log retention (delta.logRetentionDuration, default 30 days) is what bounds how far back time travel can reach.

Partitioning and Data Layout¶

[Critical] Is partitioning being applied deliberately and sparingly, sized so that each partition holds enough data to justify itself? Over-partitioning is the single most common data-layout mistake: partitioning a 200 GB table by day and by tenant produces tens of thousands of directories holding a few megabytes each, which destroys scan performance, inflates metadata, and multiplies object-storage request cost. Partition when the partition column genuinely prunes and the resulting partitions are hundreds of megabytes or larger.
[Critical] With Iceberg, is hidden partitioning understood and relied on instead of physical partition columns? Iceberg records the partition spec as transforms over source columns (identity, bucket[N], truncate[W], year, month, day, hour), and derives partition predicates from user predicates by inclusive projection -- a query filtering on ts prunes a table partitioned by day(ts) without the user naming a partition column. This eliminates the classic Hive failure where a query silently full-scans because the analyst filtered on the timestamp instead of the derived dt string column.
[Critical] Is partition evolution understood as forward-only, not a rewrite? Iceberg allows changing the partition spec without rewriting existing data. Each manifest records the spec id it was written with, and split planning converts predicates using the spec that wrote the manifest, not the current spec. The practical consequence: after an evolution, queries plan against a mix of layouts, and old data keeps the old pruning quality until it is rewritten. Delta and Hudi do not offer equivalent in-place partition evolution -- changing partitioning there generally means a rewrite.
[Recommended] For Delta, is liquid clustering used in place of partitioning for new tables? Liquid clustering is in open-source Delta (since 3.1.0; ALTER TABLE ... CLUSTER BY and OPTIMIZE FULL from 3.3), supports up to four clustering columns, and is mutually exclusive with partitioning and Z-ORDER. It avoids the cardinality and skew traps of directory partitioning.
[Recommended] Are target file sizes set consciously per format, and is the reason understood? Iceberg's write.target-file-size-bytes defaults to 512 MB; Delta's OPTIMIZE targets 1 GiB in OSS (Databricks auto-tunes by table size); Hudi's hoodie.parquet.max.file.size defaults to 120 MB with a 100 MB small-file threshold, because Hudi optimizes the write path and relies on clustering to reach larger files later. Parquet's default row-group size is 128 MB -- a data file materially smaller than one row group cannot amortize its own footer read.
[Recommended] For very large tables, is sort order or clustering configured so that queries prune within partitions? Partition pruning removes directories; sort order and column-level min/max statistics in the manifests remove files. On a well-sorted table this is often a larger win than partitioning.
[Optional] For Hudi, has the file-group layout been sized (hoodie.parquet.max.file.size, hoodie.parquet.small.file.limit, hoodie.logfile.max.size)? Hudi's writer will actively pad small files up to the small-file limit on subsequent writes, which is a genuine differentiator for streaming ingestion but only if the thresholds are set for the actual record size.

Schema and Type Evolution¶

[Critical] Is schema evolution performed through the format's evolution API rather than by rewriting tables or creating _v2 copies? All three formats support add, drop, rename, reorder, and type widening as metadata-only operations. Iceberg makes rename safe by tracking columns by field id, never by name or position -- name-tracking formats can accidentally un-delete a column when a name is reused, and position-tracking formats cannot delete a column without shifting the others.
[Critical] Are the permitted type promotions known, and is any other type change treated as a rewrite? Iceberg allows int->long, float->double, and decimal(P,S)->decimal(P',S) with larger precision; v3 adds unknown->any type and date->timestamp/timestamp_ns. Scale is never mutable, narrowing is never allowed, and long->double is not allowed. A promotion is also blocked when it would change a partition value, because bucket[N] hashes the integer 34 and the string "34" differently.
[Recommended] Is schema enforcement on the write path explicit (mergeSchema / overwriteSchema in Delta, schema-auto-update behaviour in the writing engine) rather than implicit? Silent schema drift from an upstream source is one of the most common ways a pipeline corrupts a table -- a new nullable column is harmless, a re-typed column is not.
[Optional] Where semi-structured data is landing, has the variant type been evaluated instead of storing JSON strings? Variant is GA in Delta 4.0 and is part of Iceberg spec v3. It preserves the shredding/pruning benefits of columnar storage for irregular payloads that would otherwise be an opaque string column.

Row-Level Mutation: Copy-on-Write vs Merge-on-Read¶

[Critical] Has the copy-on-write vs merge-on-read decision been made per table based on the write/read ratio, rather than accepting a global default? Copy-on-write rewrites every data file touched by an update, so writes are expensive and reads are clean. Merge-on-read writes delete/change records alongside the base files, so writes are cheap and every read pays a merge cost that grows until compaction runs. High-frequency small updates with tolerant read SLAs favour MoR; daily restatement with strict interactive read SLAs favours CoW.
[Critical] For Iceberg, are write.delete.mode, write.update.mode, and write.merge.mode set explicitly? All three default to copy-on-write, which is often the wrong choice for a CDC target. Note also that they are honoured by Spark but not uniformly by other engines -- Athena SQL, for example, documents that it always uses merge-on-read with positional deletes and silently ignores these properties. Cross-engine standardization on these settings does not work.
[Critical] For Hudi, is a compaction strategy configured for every merge-on-read table, and has it been verified to actually run? hoodie.compact.inline and hoodie.compact.schedule.inline both default to false. Compaction runs automatically for the streaming write models (Hudi Streamer in continuous mode, Flink, Spark Streaming); a plain batch Spark DataSource write to a MoR table runs no compaction at all unless configured. Uncompacted MoR tables degrade until reads fail.
[Recommended] Is the difference between Iceberg's position deletes and equality deletes understood by whoever operates the pipeline? Position deletes name a file and a row offset; equality deletes name predicate values and are what Flink's upsert path emits. The applicability rules differ: position deletes and deletion vectors apply when the data sequence number is less than or equal to the delete's, so they can delete rows added in the same commit; equality deletes apply only when strictly less. An equality delete written with an unpartitioned spec applies globally and is expensive to evaluate.
[Recommended] If Iceberg spec v3 is in play, is the shift to deletion vectors accounted for? v3 replaces position delete files with Puffin-encoded roaring bitmaps, at most one per data file per snapshot, and prohibits writing new position delete files. Superseded deletion-vector blobs are not required to be rewritten out of their Puffin files, so orphan-file cleanup is what actually reclaims them.
[Recommended] For Hudi, is the index type chosen to match the update pattern (BLOOM/GLOBAL_BLOOM, SIMPLE, BUCKET, record-level index) and is the cost model understood? A non-global index makes lookup cost proportional to the number of records updated; a global index makes it proportional to table size. The default is SIMPLE on Spark and Java and FLINK_STATE on Flink. Note that RECORD_INDEX was deprecated in Hudi 1.1 in favour of the newer record-level-index configuration.
[Optional] Where GDPR/CCPA erasure produces frequent scattered point deletes, are deletion vectors (Delta, Iceberg v3) enabled so that a single-row delete does not rewrite a 512 MB file? This is one of the clearest cases where MoR mechanics reduce cost by orders of magnitude.

Snapshots, Time Travel, and Retention¶

[Critical] Is the retention window set from an actual requirement (rollback horizon, audit obligation, reproducible reporting) rather than left at defaults? Iceberg's history.expire.max-snapshot-age-ms defaults to 5 days with min-snapshots-to-keep of 1; Delta's delta.deletedFileRetentionDuration defaults to 7 days and delta.logRetentionDuration to 30 days. Time travel beyond the retention window fails -- the metadata may still name a snapshot whose files have been deleted.
[Critical] Is it understood that every retained snapshot pins its data files, so retention is a storage-cost decision as much as a recovery decision? A table with a long retention window, a high commit rate, and regular compaction can hold several multiples of its logical size, because compaction temporarily doubles storage until the pre-compaction snapshots expire.
[Critical] Is lowering Delta's VACUUM retention below the default understood as dangerous rather than as an optimization? Delta's own documentation warns that if VACUUM removes files still referenced by a concurrent reader or in-flight writer, readers fail and tables can be corrupted. The retention-duration safety check exists for this reason and should not be routinely disabled.
[Recommended] Is time travel actually being used for something (audit reproduction, incident rollback, MERGE idempotency verification, ML training-set reproducibility), or is it being paid for by default? Retention has a real bill. If nothing consumes it, shorten it.
[Recommended] For Hudi, are the cleaner and archival services configured and running? hoodie.clean.automatic defaults to true with KEEP_LATEST_COMMITS and 10 commits retained; archival keeps 20-30 commits on the active timeline. Without archival the active timeline -- which is consulted on every read and write -- grows unboundedly and every operation slows. Note that the cleaner config keys were renamed in Hudi 1.x (hoodie.cleaner.policy -> hoodie.clean.policy, hoodie.cleaner.commits.retained -> hoodie.clean.commits.retained); the old keys silently do nothing on 1.x.

Maintenance and the Small-File Problem¶

[Critical] Is there a scheduled, monitored, owned maintenance job for every mutable table -- compaction, snapshot expiry, and orphan-file cleanup -- rather than an assumption that the format handles it? This is the single most common operational failure with open table formats. All three formats are libraries plus metadata conventions; none of them run background jobs on their own. Managed offerings (AWS S3 Tables, Glue table optimizers, Databricks predictive optimization) exist precisely because so many teams skip this.
[Critical] Is the small-file cost understood as three separate costs rather than one? (1) Request cost and latency -- each file needs a LIST/GET plus a footer read, and S3 delivers roughly 5,500 GET/HEAD per second per prefix with first-byte latency around 100-200 ms for small objects; (2) metadata bloat -- more files means more manifest entries, longer query planning, and, on Glue, an eventual hard 50 MB metadata ceiling; (3) worse compression -- small files compress less effectively and carry proportionally larger footers.
[Critical] Is orphan-file cleanup scheduled separately from snapshot expiry, and is the distinction understood? Expiring a snapshot unlinks files from the metadata tree; it does not necessarily delete every unreferenced object. Iceberg's remove_orphan_files (default older_than of 3 days) is what removes files that were never committed -- failed writes, abandoned compactions, superseded Puffin blobs. Run it with a generous age threshold: an aggressive threshold can delete files belonging to an in-flight write.
[Critical] For Iceberg, is metadata-file cleanup enabled? write.metadata.delete-after-commit.enabled defaults to false, with write.metadata.previous-versions-max of 100. With defaults, a table with 100 commits retains 10 tracked metadata files and 90 untracked ones -- and turning the property on afterwards does not clean them, because they are already untracked. Enable it at table creation.
[Recommended] Are the compaction thresholds tuned rather than left at defaults that may never trigger? Iceberg's rewrite_data_files defaults to a 512 MB target with a 75%-180% acceptance band, min-input-files of 5, and a delete-file-threshold that is effectively disabled -- so a table accumulating delete files may never be selected for rewrite on file-count grounds alone.
[Recommended] Are rewrite_manifests and, where applicable, delete-file compaction run in addition to data-file compaction? Data-file compaction alone can leave dangling delete files and a manifest tree that has grown wide, both of which show up as query planning time rather than scan time -- a symptom that is easy to misattribute to the engine.
[Recommended] Where a managed maintenance service is available and the tables are managed tables, is it enabled instead of hand-rolled jobs? AWS Glue offers table optimizers for Iceberg (compaction, snapshot retention, orphan deletion) billed per DPU-hour; S3 Tables performs maintenance inside the table bucket; Databricks predictive optimization runs OPTIMIZE/VACUUM/ANALYZE on Unity Catalog managed tables only -- not external tables -- and notably does not run ZORDER as part of it.
[Optional] Is there a monitoring signal on file count per partition, average file size, delete-file count, and metadata size, so that maintenance failure is detected before users notice slow queries? These are cheap to compute from the format's own metadata tables and are the leading indicator for every failure mode above.

Engine Interoperability¶

[Critical] Has the read/write/DML matrix been verified for every engine in scope, against current documentation, rather than assumed from "engine X supports Iceberg"? The distinction that matters is not read vs no-read; it is read vs write vs full row-level DML vs maintenance. A representative snapshot of Iceberg support in mid-2026: Spark has full DML (requires the Iceberg SQL extensions); Flink writes but has no MERGE/UPDATE/DELETE, only upsert via equality deletes; Trino has full DML on v1/v2 with v3 experimental; Snowflake supports managed and externally-managed tables but does not support equality deletes; BigQuery managed Iceberg tables support DML with a limit of one concurrent mutating statement per table; BigLake external Iceberg tables are read-only; Athena supports v2 only and always uses merge-on-read; Redshift can write but has no time travel; DuckDB can write only through an attached REST catalog.
[Critical] Is exactly one engine designated as the writer of record for each table, with the others read-only? Multi-writer across heterogeneous engines is where the differences above turn into corruption or silent no-ops -- different default modes, different delete encodings, different maintenance assumptions. Concurrent multi-engine writes should be a deliberate, tested configuration, never an emergent one.
[Recommended] Where a warehouse reads lake tables, is it clear which side owns the table -- warehouse-managed (the warehouse writes and maintains, other engines read) or externally managed (an external engine writes, the warehouse reads)? The two have materially different capability sets in every warehouse that offers both, and the difference is easy to lose in an architecture diagram.
[Recommended] Have the delete-encoding gaps been checked specifically? Several engines that read Iceberg do not support equality deletes, which are exactly what a Flink upsert pipeline produces. A Flink-written CDC table can be perfectly valid and still be unreadable by a downstream warehouse.
[Optional] If the table must be exposed to a specific vendor ecosystem, has the vendor's own interop mechanism been evaluated before building a translation layer? Examples include Databricks Unity Catalog's Iceberg REST endpoint, Snowflake catalog-linked databases, and Microsoft Fabric's metadata virtualization (which presents Iceberg tables with a virtual Delta log and vice versa, for Iceberg v2 only). These are generally more reliable than self-managed metadata translation.

Cost and Operations¶

[Critical] Does the cost model include compaction compute, not just storage and query? Compaction is the recurring hidden cost of an open table format. A table receiving frequent small writes can spend more on rewriting itself than on serving queries, and the cost scales with commit frequency rather than with data volume.
[Critical] Are object-storage request costs modelled, not just capacity? At roughly $0.005 per 1,000 PUTs and $0.0004 per 1,000 GETs on S3 in us-east-1, a table with tens of millions of small objects and a high query rate generates a request bill that can rival its storage bill.
[Recommended] If AWS S3 Tables is under consideration, are its two extra billing dimensions accounted for? Beyond storage (published at $0.0265/GB-month for the first 50 TB in us-west-2), S3 Tables bills object monitoring at $0.025 per 1,000 objects per month and compaction at $0.002 per 1,000 objects plus $0.005/GB. A small-file-heavy table is therefore charged twice for its file count -- once to be watched, once to be fixed. Ten million small objects is roughly $250/month in monitoring alone.
[Recommended] Is Glue Data Catalog metadata cost accounted for on partition-heavy lakes? The first million catalog objects are free, then roughly $1.00 per 100,000 objects per month, and every partition is an object. An over-partitioned lake pays for its over-partitioning three times: in scan performance, in catalog storage, and in compaction.
[Optional] Is there a periodic review of tables that have accumulated snapshots, delete files, or partitions far beyond what their query patterns justify? Table-format cost problems are almost always concentrated in a handful of pathological tables rather than spread evenly.

Why This Matters¶

The table format is the most durable decision in a lakehouse. Compute engines get swapped every few years -- a team moves from EMR to Databricks, adds Trino, replaces one warehouse with another -- and each of those migrations is survivable because the data did not move. The table format is what makes that true, and it is also what breaks it: a lake standardized on a format that a future engine cannot write is a lake that has to be rewritten. This is why the format decision deserves an ADR while the engine decision often does not.

The failure mode that actually shows up in production is not choosing "wrong" -- all three formats are competent -- but treating the format as a file layout rather than a database that needs an operator. None of these formats runs anything in the background. Snapshots accumulate until storage cost triples. Small files accumulate until query planning takes longer than query execution. Delete files accumulate until merge-on-read reads become slower than the copy-on-write writes they were meant to avoid. Metadata accumulates until a catalog rejects the table outright. Every one of these is a scheduled job that nobody owned, and every one of them presents first as "the platform got slow" rather than as an alert.

Partitioning is where inexperience is most expensive and most invisible. The intuition that more partitions means better pruning is wrong past a threshold, and the threshold is lower than most teams expect. An over-partitioned table has poor scan performance because it reads thousands of tiny files, poor planning performance because the manifest tree is wide, high catalog cost because every partition is a metadata object, and high compaction cost because the compactor has more work and less material to work with. Iceberg's hidden partitioning and Delta's liquid clustering both exist to take this decision away from users who keep getting it wrong; using them is usually better than getting it right by hand.

Interoperability claims deserve more scepticism than they usually get. "Engine X supports Iceberg" is true of nearly every engine and tells you almost nothing. What matters is whether that engine can write, whether it can perform row-level DML, whether it honours the table's configured delete mode, whether it can read the delete encoding a different engine produces, and whether it can run maintenance. Those answers differ per engine, per format version, and per month. An architecture that assumes uniform capability across engines will discover the gaps at the point of highest cost -- after the data is written.

Finally, the consolidation around Iceberg is real but should be understood for what it is. Snowflake donated Polaris to the Apache Software Foundation, where it became a top-level project; Databricks acquired Tabular and publicly committed to working toward interoperability between Iceberg and Delta at the format level, while noting that this would take years; AWS shipped S3 Tables as a first-class Iceberg storage type; Google, Confluent, and Microsoft all built Iceberg paths. That is strong evidence about optionality -- Iceberg is the format least likely to strand you. It is not evidence that Iceberg is technically superior for a given workload, and it does not mean format convergence has shipped. Delta remains the better-integrated choice inside Databricks, and Hudi remains measurably better at high-frequency upserts.

Common Decisions (ADR Triggers)¶

general/data-analytics.md holds the summary ADR: Open Table Format Selection. The triggers below are the second-order decisions that follow it and each deserve their own record.

Table format -- Iceberg for maximum engine optionality and an open catalog specification vs Delta Lake for the deepest Databricks integration and the most mature single-vendor tooling vs Hudi for upsert-dominated CDC and streaming ingestion where record-level indexing avoids whole-table joins. Decide from the write side; document the engines that will be permitted to write.
Iceberg spec version -- v2 for the widest engine compatibility today vs v3 for deletion vectors, row lineage, variant and geospatial types, accepting that engine support lags and that some engines will refuse or degrade. Note the library still defaults to v2.
Catalog -- Iceberg REST / Polaris for portability and an open specification vs a cloud-native catalog (Glue, Unity Catalog, Snowflake Horizon) for integrated governance and lower operational overhead vs Nessie for git-style branching workflows vs Hive Metastore only as a migration source. The catalog owns commit atomicity and access control, so this is a correctness and security decision, not just a metadata store.
Managed vs self-operated table maintenance -- managed (S3 Tables, Glue table optimizers, Databricks predictive optimization) for lower operational risk at a metered cost vs self-scheduled Spark maintenance jobs for control and lower unit cost, accepting that they will be forgotten. Note that managed optimization usually applies only to managed tables, not external ones.
Copy-on-write vs merge-on-read, per table class -- CoW for read-latency-critical serving tables with infrequent restatement vs MoR for high-frequency mutation with compaction on a schedule. This is a per-table decision that should be encoded in table-creation templates rather than left to whoever writes the first pipeline.
Partitioning strategy -- Iceberg hidden partitioning with transforms vs Delta liquid clustering vs explicit directory partitioning vs no partitioning with sort order and file-level statistics. For tables under roughly a terabyte, "no partitioning, good sort order" is more often correct than teams expect.
Snapshot retention window -- short (days) to minimize storage and compaction overhead vs long (weeks or months) for audit reproducibility and rollback, priced explicitly against the storage the retained snapshots pin.
Writer topology -- single writer of record per table with read-only consumers (simple, safe, the default recommendation) vs coordinated multi-engine writes (necessary for some architectures, requires verified compatibility of delete encodings and delete modes across every writer).
Format translation -- accept a single format everywhere vs run UniForm or XTable to expose a second format read-only vs maintain genuinely duplicated tables. Translation is one-way and read-only; treat any plan that requires writing through the translation layer as unsupported.
Migration from Hive tables -- in-place migration (Iceberg add_files / migrate, Delta CONVERT TO DELTA) for speed and no data movement vs full rewrite for a clean layout, correct file sizes, and a chance to fix partitioning. In-place migration inherits the old table's small-file and partitioning problems.

Reference Architectures¶

Iceberg on AWS with Glue Data Catalog¶

Landing data in S3 (raw, immutable) -> Spark on EMR or Glue ETL writes Iceberg tables with Glue Data Catalog as the Iceberg catalog -> Athena and Redshift Spectrum read; Spark writes. Glue table optimizers run compaction, snapshot retention, and orphan-file deletion per table. Lake Formation grants column- and row-level permissions over the catalog. Watch the 50 MB catalog metadata ceiling on high-commit tables. See providers/aws/glue.md, providers/aws/athena.md, providers/aws/lake-formation.md.

Iceberg with an open REST catalog, multi-engine¶

S3/ADLS/GCS object storage -> Iceberg tables registered in a REST catalog (Polaris or a vendor REST endpoint) -> Spark as writer of record for batch, Flink for streaming upserts into a separate set of MoR tables, Trino for interactive SQL, and a warehouse attached read-only through the same REST catalog. Maintenance runs as scheduled Spark jobs (rewrite_data_files, expire_snapshots, remove_orphan_files, rewrite_manifests). The REST catalog is the single enforcement point for credentials and grants. Verify equality-delete support in every reader before letting Flink write.

Delta Lakehouse on Databricks¶

ADLS Gen2 or S3 -> Auto Loader / Delta Live Tables write Delta tables registered in Unity Catalog -> liquid clustering instead of partitioning -> predictive optimization runs OPTIMIZE/VACUUM/ANALYZE on managed tables -> Databricks SQL and BI tools read. External engines read through Unity Catalog's Iceberg REST endpoint or, for older setups, UniForm-generated Iceberg metadata (read-only). See providers/databricks/data-platform.md.

Hudi CDC sink¶

Debezium or a CDC service -> Kafka -> Hudi Streamer (or Flink) writes merge-on-read Hudi tables keyed by primary key, with a record-level or bloom index sized to the update pattern -> async compaction, cleaning, clustering, and archival run continuously -> read-optimized queries serve BI while snapshot queries serve freshness-sensitive consumers. Verify that compaction is genuinely scheduled: batch DataSource writes do not compact by default.

Iceberg on S3 Tables¶

S3 table buckets hold Iceberg tables with maintenance (compaction, snapshot expiry, unreferenced file removal) performed by the service. Integrated with Glue/Lake Formation for governance and queryable from Athena, EMR, Redshift, and Spark. Trades per-object monitoring and compaction fees for the elimination of self-managed maintenance jobs -- attractive when the alternative is that nobody schedules them. Note that creating a user-defined branch or tag disables automated snapshot management for that table.

Reference Links¶

Apache Iceberg Table Spec -- format versions, metadata/manifest structure, partition transforms, type promotion rules, and delete-file semantics
Iceberg Configuration Reference -- table properties including write.target-file-size-bytes, write.delete.mode, and history.expire.* defaults
Iceberg Maintenance -- snapshot expiry, orphan file removal, compaction, and manifest rewriting
Iceberg Spark Procedures -- rewrite_data_files, expire_snapshots, remove_orphan_files, rewrite_manifests and their arguments
Iceberg Schema Evolution -- safe schema and partition evolution, and the field-id mechanism
Iceberg Partitioning -- hidden partitioning, inclusive projection, and partition evolution
Iceberg Branching and Tagging -- write-audit-publish, branch vs tag schema semantics, and ref retention
Iceberg REST Catalog Spec -- the open catalog API including commit requirements and multi-table transactions
Iceberg Puffin Spec -- the blob format used for deletion vectors and statistics in spec v3
Iceberg Multi-Engine Support -- supported Spark and Flink versions per Iceberg release
Delta Lake Protocol -- transaction log actions, protocol versions, table features, deletion vectors, and catalog-managed commits
Delta Lake Storage Configuration -- LogStore implementations per cloud and the S3 multi-writer DynamoDB requirement
Delta Lake Table Utilities -- VACUUM, history, and table detail commands
Delta Lake Optimizations -- OPTIMIZE, Z-ORDER, and file-size configuration
Delta Lake Clustering -- liquid clustering in open-source Delta, column limits, and OPTIMIZE FULL
Delta UniForm -- generating Iceberg/Hudi metadata from Delta tables and the read-only constraint
Delta Lake Versioning -- protocol reader/writer versions and table features
Apache Hudi Table Types -- copy-on-write vs merge-on-read, file groups and file slices, query types
Apache Hudi Timeline -- instants, action types, completion-time semantics, and the LSM timeline
Apache Hudi Indexes -- index types, the metadata table, and the upsert cost argument
Apache Hudi Configurations -- compaction, cleaning, archival, and file-sizing defaults
Apache Hudi Tech Specs -- the storage format specification
Trino Iceberg Connector -- supported spec versions, DML, and maintenance procedures from Trino
AWS Glue Iceberg REST Catalog -- the Glue-hosted Iceberg REST endpoint and supported operations
Amazon S3 Tables Maintenance -- managed compaction, snapshot management, and unreferenced file removal
Databricks Iceberg Interoperability -- Unity Catalog managed Iceberg tables and the Iceberg REST endpoint
Apache Polaris -- the open-source Iceberg REST catalog implementation
Project Nessie -- git-style branching and tagging for Iceberg catalogs
Apache XTable -- incubating cross-format metadata translation between Iceberg, Delta, and Hudi