For most of the last decade, choosing a big data platform meant choosing a storage format, and choosing a storage format meant choosing a vendor. Data written by one engine was awkward or impossible to use from another, so every new tool came with a copy job, a sync process, and another place for numbers to drift. In 2026 that bargain has largely broken down. Apache Iceberg has become the common table format that Snowflake, Databricks, AWS, Google Cloud and the open-source engines all read and write, and the version 3 specification is now generally available across the platforms most enterprises already pay for.
That is good news, but it creates a new set of decisions. Iceberg v3 introduces features that change how updates, semi-structured data and change data capture work, and the upgrade is one-way. This article explains what v3 actually changes, why it matters for big data workloads, and how to plan an adoption that improves cost and flexibility instead of creating a fresh round of incompatibilities.
Why Open Table Formats Won the Format War
A table format is the metadata layer that sits on top of plain data files, usually Parquet, in object storage. It records which files belong to a table, which snapshot is current, how the table is partitioned, and what its schema looks like. Without that layer, object storage is just a pile of files; with it, many engines can safely read and write the same data with transactional guarantees.
Iceberg, Delta Lake and Apache Hudi all solved this problem in slightly different ways. What tipped the market toward Iceberg was not a single feature but breadth of adoption. Its specification is engine-neutral, it is governed by the Apache Software Foundation, and its REST catalog protocol gave vendors a standard way to expose tables to outside engines. By the time of the Iceberg Summit in San Francisco in April 2026, every major cloud data platform was participating, and coverage of the event described v3 support as the fastest adoption cycle the project has seen for a spec version.
For big data teams, the practical consequence is simple: you can now store data once, in your own cloud account, and choose engines per workload. Spark or Flink for heavy transformation, a warehouse engine for business intelligence, DuckDB or Trino for ad hoc exploration, and a machine learning stack reading the same tables. The format is no longer the lock-in point. The catalog and the compute are.
What Iceberg v3 Actually Adds
The v3 specification was finalized by the Apache Iceberg project in 2025, and engine support arrived through 2026. Snowflake reached general availability for v3 tables in May 2026, Databricks moved from a Runtime 18.0 public preview in April to general availability in June, and Amazon S3 Tables added deletion vectors and row lineage in late 2025 and VARIANT support in July 2026. Apache Iceberg 1.11.0, released in May 2026, made Spark 4.1 and Flink 2.1 the default build targets and added server-side scan planning in the REST catalog. The headline features are:
- Deletion vectors. Instead of rewriting whole files or piling up positional delete files, v3 marks deleted rows in a compact per-file bitmap stored in a Puffin file. Updates and deletes become cheaper to write and faster to read.
- Row lineage. Each row carries a stable row ID and a last-updated sequence number, so incremental pipelines can ask the table what changed since a given snapshot instead of diffing data themselves.
- VARIANT type. A binary encoding for semi-structured data such as JSON events, with the option to shred frequently queried fields into typed Parquet columns for pruning and speed.
- Default column values. New columns can declare a default, so older files return that value on read and no backfill job is needed.
- Geospatial types. Native geometry and geography types make location data a first-class citizen of the table format.
- Nanosecond timestamps. Useful for telemetry, trading and IoT data where microsecond precision was a limitation in v2.
Why deletion vectors matter most
Big data platforms were originally designed for append-only workloads. Modern use cases are not append-only. GDPR erasure requests, late-arriving corrections, slowly changing dimensions and CDC streams from operational databases all generate updates and deletes. In v2, teams chose between copy-on-write, which rewrites entire files and is expensive for small changes, and merge-on-read with positional or equality delete files, which is cheap to write but degrades read performance until compaction runs. Deletion vectors narrow that tradeoff considerably. Vendor benchmarks cited by AWS for EMR report large improvements in DML speed versus v2, but treat any single figure as workload-dependent and test with your own data.
Why row lineage matters for CDC and AI
Row lineage is less visible but arguably more strategic. Many organizations maintain fragile incremental logic: watermark columns, hash comparisons, or full-table diffs to work out what changed. With row lineage, the table itself records the change history at row level. That simplifies downstream materializations, feature pipelines for machine learning, and audit requirements where you must show when a record was last modified.
Interoperability Is Real, but Not Uniform
It is tempting to read “GA everywhere” as “identical everywhere.” It is not. Each platform implements v3 on its own schedule and with its own defaults. For example, Databricks defaults new Iceberg tables to v3, while Snowflake requires you to create v3 tables explicitly. During Snowflake’s preview, externally managed tables were limited to v2 writes and some delete modes were unavailable. These gaps close over time, but at any given moment your engine matrix will have rough edges.
The single most important caveat is that a v3 upgrade cannot be rolled back. Once a table’s format version is raised, every engine that touches it must support v3, including the forgotten ones: the nightly export job running an old Spark version, the BI extract built on a vendor connector, the data science notebook pinned to an older library. Upgrading a table that a v2-only reader depends on will break that reader.
Iceberg removes format lock-in, but it does not remove the need for an engine compatibility matrix. Treat every v3 upgrade as a contract change for every reader and writer of that table.
A Decision Framework: Which Tables Should Move to v3 First
Not every table benefits equally. Use these criteria to prioritize:
- High update or delete volume. CDC-fed tables, customer and account dimensions, and any table subject to privacy erasure gain the most from deletion vectors.
- Semi-structured payloads. Event and log tables that currently store JSON as strings benefit from VARIANT, especially if analysts repeatedly extract the same handful of fields.
- Incremental consumers. Tables feeding many downstream incremental models or feature stores benefit from row lineage.
- Controlled reader set. Prefer tables whose readers are known and all v3-capable. Shared, widely consumed tables should move later, after the long tail of readers is upgraded.
- Low regulatory friction. Start where a failed upgrade would be inconvenient, not where it would interrupt financial close or regulatory reporting.
It also helps to frame the decision for business stakeholders in terms they recognize. Deletion vectors translate into faster turnaround on privacy requests and fresher customer data. VARIANT translates into fewer brittle parsing jobs and quicker onboarding of new event sources. Row lineage translates into simpler audits and cheaper incremental processing. If a candidate table cannot be linked to at least one of these outcomes, it probably belongs in a later wave. Writing this justification down for each wave also gives you a baseline to measure against after the upgrade, which makes the next round of prioritization far easier to defend.
Conversely, leave large, append-only, rarely updated history tables on v2 until there is a clear reason. They gain little from deletion vectors, and they are often the tables with the most diverse readers.
A Practical Upgrade Playbook
A disciplined rollout takes weeks, not quarters, if you prepare the inventory first. The steps below assume a platform team responsible for a shared lakehouse with several engines.
Step 1: Build the reader and writer inventory
For each candidate table, list every engine, library and connector that reads or writes it, including versions. Catalog audit logs and query history are the fastest source. Pay special attention to scheduled jobs owned by other teams and to third-party tools connecting through service accounts.
Step 2: Pin library versions
Standardize on a known Iceberg library version across Spark, Flink and custom services, such as the 1.11 line, and on engine runtimes that support v3. Mixed versions are the most common source of subtle failures, especially around delete handling.
Step 3: Pilot on a cloned table
Create a copy of a representative table, upgrade it, and replay a realistic week of writes and reads. Measure write latency, read latency, file counts and storage growth. Confirm that every engine in the inventory can read the result and that row counts and aggregates match the v2 original.
Step 4: Convert JSON columns deliberately
Moving a string column of JSON to VARIANT is a schema change with downstream impact. Identify the fields queried most often and test shredding, then update views and models that parse JSON so they use variant access instead. Do this table by table rather than as a big-bang change.
Step 5: Revisit maintenance jobs
Compaction, snapshot expiration and orphan file cleanup still matter in v3. Deletion vectors reduce the urgency of compaction for update-heavy tables but do not eliminate it. If you use a managed service such as S3 Tables or Snowflake-managed Iceberg storage, confirm which maintenance tasks the service performs and which remain yours.
Step 6: Upgrade in waves with a freeze window
Upgrade tables in small groups, announce a short freeze on schema changes for each wave, and keep an operational runbook. Because downgrade is impossible, your fallback is to restore readers, not to revert the table.
Pitfalls We See Most Often
- Forgetting the long tail of readers. A single legacy extract can block an upgrade or break silently after it.
- Catalog mismatch. Engines that talk to different catalogs for the same table, or bypass the catalog by reading metadata paths directly, can see inconsistent states.
- Treating VARIANT as a dumping ground. VARIANT makes semi-structured data cheaper to store and query, but it does not replace a data contract. Agree on the fields that matter.
- Skipping maintenance budgets. Open formats shift some operational work from the vendor to you. Small files and stale snapshots still cost money.
- Assuming feature parity. Check each engine’s documentation for v3 limitations before promising features to downstream teams.
A Worked Example: An Event Platform Moving to v3
Consider a mid-sized e-commerce company with three main Iceberg tables: a clickstream table receiving JSON events from a streaming pipeline, a customer dimension fed by CDC from the operational database, and a ten-year order history table used by finance. Spark handles ingestion, a cloud warehouse serves BI, and data scientists use Python notebooks.
The platform team starts with the customer dimension. It receives frequent updates and must honor deletion requests within a fixed period, so deletion vectors offer a clear benefit. The inventory reveals one surprise: a marketing export job runs an old Spark image. The team upgrades that image first, pilots the v3 table on a clone for a week, compares row counts and key aggregates, and then upgrades the production table during a quiet window.
The clickstream table is next. Analysts repeatedly extract the same handful of fields from the event payload, so the team migrates the payload column to VARIANT with those fields shredded. They rewrite the five most-used views to use variant access and leave a compatibility view for older dashboards during the transition.
The order history table stays on v2. It is append-only, read by many finance tools, and would gain little. The team documents the decision and sets a review date. The outcome is not a dramatic platform rewrite; it is a targeted set of changes where the new capabilities pay for themselves, with no broken readers.
Future Outlook: The Catalog Becomes the Control Plane
With the format question largely settled, competition has moved to the catalog and to managed table services. Apache Polaris graduated to a top-level Apache project in February 2026, the open-source Unity Catalog added full Iceberg REST support, and every hyperscaler now offers a managed Iceberg catalog. Expect the next two years to focus on governance that travels with the data across engines, fine-grained access control enforced through credential vending, and automatic optimization services that make open tables feel as low-maintenance as a proprietary warehouse.
For big data teams, the strategic posture is clear: store data in open tables in storage you control, keep the engine layer replaceable, and invest in the catalog and governance layer, because that is where lock-in and value now concentrate.
Conclusion
Iceberg v3 is the most meaningful update to the open table format since v2 introduced row-level deletes. Deletion vectors, row lineage and VARIANT address real pain in update-heavy, CDC-driven and semi-structured workloads. But the upgrade is irreversible and support is uneven across engines, so success depends on inventory, pinned versions, pilots and waves rather than a blanket switch. Start with the tables that gain the most and have the smallest set of readers, measure the results, and expand from there.
How Hanoi Technology Can Help
Hanoi Technology’s data engineers help teams inventory their Iceberg estate, pilot v3 upgrades and run open lakehouse platforms as a project, dedicated team or managed service.