Two years ago, lakehouse debates were about table formats: Iceberg, Delta Lake or Hudi. That argument has largely been settled by interoperability, with Apache Iceberg readable and writable across every major platform and Delta tables bridged through compatibility features. The decision that now shapes a lakehouse’s future is the catalog: the service that tracks which tables exist, where their current metadata lives, who may access them and how engines get credentials to read the underlying files.
Industry commentators have called the contest between catalogs the “catalog wars,” and in 2026 it intensified. Apache Polaris graduated to a top-level Apache project, open-source Unity Catalog added full Iceberg REST support, Snowflake moved its external catalog story toward Horizon, and Google renamed and expanded its lakehouse catalog. This guide explains what a catalog really controls, compares the main options, and offers a practical way to choose.
Why the Catalog Became the Control Plane
In an open lakehouse, data files sit in object storage and table formats describe them. But engines still need to answer three questions before they can run a query: which metadata file represents the current state of this table, is this user allowed to read or change it, and what credentials should be used to fetch the files. The catalog answers all three.
That makes the catalog the natural place for governance. Access policies, audit logs, lineage, data classification and table maintenance all hang off it. It also makes the catalog the new lock-in point. If every engine must go through a single vendor’s catalog, and that catalog only fully cooperates with that vendor’s compute, the openness of the file format matters much less in practice.
The Iceberg REST catalog specification is the reason this competition is healthy rather than fragmented. It defines a standard HTTP API for listing, loading and committing tables, and increasingly for vending temporary storage credentials and planning scans on the server. Any engine that speaks the protocol can work with any catalog that implements it, at least for the core operations.
The Main Contenders
Apache Polaris
Polaris is an open-source implementation of the Iceberg REST catalog, originally created by Snowflake with community contributors including Dremio, and governed by the Apache Software Foundation. It graduated to a top-level project in February 2026 and has released steadily since, with features such as native credential vending for AWS, Google Cloud and Azure, role-based access control and metrics persistence. It focuses on Iceberg and on multi-engine, multi-cloud neutrality. You can self-host it or consume it through commercial services built on it.
Unity Catalog
Unity Catalog exists in two forms: the managed service at the heart of the Databricks platform, and an open-source project under the Linux Foundation’s LF AI and Data umbrella. The managed version governs not only tables but also files, machine learning models, functions and metric definitions, with lineage and fine-grained access control. The open-source project has moved quickly in 2026, adding catalog-managed Delta commits, metric views and, in October, full Iceberg REST table support. Its distinguishing strength is covering both Delta and Iceberg and extending governance beyond tables to AI assets.
Snowflake Horizon Catalog
Snowflake’s Horizon Catalog exposes an Iceberg REST endpoint so external engines such as Spark or Trino can read Snowflake-managed Iceberg tables. Snowflake has stopped accepting new Open Catalog accounts, steering customers toward Horizon, and has signaled billing for external access in the second half of 2026. One caveat noted by practitioners: policies such as row access and masking may not be enforced on every external access path, so verify how protected tables behave before relying on them.
AWS Glue Data Catalog and S3 Tables
On AWS, the Glue Data Catalog serves the Iceberg REST API with SigV4 authentication and works with both general-purpose S3 buckets and S3 Tables, which provide managed Iceberg storage with automatic maintenance. Access control is handled through IAM and Lake Formation. It is serverless and well integrated with Athena, EMR and Redshift, but cross-cloud engines must manage AWS credentials.
Google Cloud Lakehouse runtime catalog
Google renamed its BigLake metastore in April 2026 and offers a generally available Iceberg REST endpoint for open engines, alongside BigQuery-managed Iceberg tables with credential vending for table-level access control. It suits organizations whose analytics center on BigQuery while using open engines for other workloads.
Comparison at a Glance
- Formats: Polaris, Horizon, Glue and Google’s catalog focus on Iceberg; Unity Catalog spans Delta and Iceberg.
- Governance scope: Unity Catalog extends to files, models and metrics; Polaris focuses on tables, namespaces and roles; cloud catalogs integrate with their own identity and policy systems.
- Neutrality: Polaris is foundation-governed and designed for multi-cloud; open-source Unity Catalog is foundation-governed but most mature in its Databricks form; cloud catalogs are proprietary services tied to their provider.
- Credential vending: All major options now vend scoped, temporary credentials in some form, which removes the need to hand engines long-lived storage keys.
- Operations: Managed cloud catalogs minimize operational effort; self-hosted Polaris or open-source Unity Catalog require you to run, secure and upgrade a service.
- Cost model: Open-source catalogs are free to run but cost operations time; managed catalogs charge by requests, capacity or platform subscription.
Choose the catalog for the next five years of engines, not the one you use today. Whoever controls the catalog controls governance, and governance is the hardest thing to move later.
A Practical Selection Method
The right choice depends far more on your engine mix and cloud strategy than on feature checklists. Work through these questions in order.
- Which formats must you govern? If you have substantial Delta Lake estates alongside Iceberg, Unity Catalog is the most natural fit. If you are Iceberg-only, every option is viable.
- What is your primary compute platform? Organizations centered on Databricks gain most from Unity Catalog; those centered on Snowflake from Horizon; single-cloud shops from their provider’s native catalog.
- How many engines and clouds must be first-class? If several engines across more than one cloud must read and write with equal standing, a neutral catalog such as Polaris reduces dependence on any one vendor.
- Where must policy be enforced? Map which engines must enforce row-level security and masking. Some catalogs enforce policies only within their own engines, which matters for regulated data.
- Do you need governance beyond tables? If models, vector indexes, files and metrics must be governed together, favor a catalog that covers those objects.
- How much do you want to operate? Self-hosting gives control but requires high availability, backups, upgrades and security hardening.
Look at total cost of ownership
Price comparisons between catalogs are often misleading. A free open-source catalog still needs a highly available deployment, a backing database, monitoring, security patching and people who understand it. A managed catalog may charge per request or for external access, which can grow quickly when many engines poll metadata frequently. Estimate request volumes from current engine behavior, include the engineering hours for operations, and consider the cost of a future migration if commercial terms change.
Federation is a valid answer
Many enterprises will run more than one catalog: a platform catalog for one major engine and a neutral or cloud catalog for others, with federation or synchronization between them. This is workable if one catalog is clearly the system of record for each table and others reference it, rather than both accepting writes. Dual writers are a reliable way to corrupt table state.
Implementation Playbook
- Inventory tables, engines and access paths. Identify which engines read and write which tables today, and how they authenticate. Direct path-based access that bypasses any catalog is the first thing to eliminate.
- Define ownership and naming. Agree on catalog, namespace and table naming conventions aligned to domains, and assign an owner to each namespace.
- Design the permission model. Map business roles to catalog roles, decide how service principals are scoped, and plan how policies propagate to engines.
- Enable credential vending. Remove long-lived storage keys from jobs and notebooks; engines should receive short-lived, table-scoped credentials from the catalog.
- Register or migrate tables. Register existing Iceberg tables by pointing the new catalog at their current metadata, test reads from every engine, then switch writers one at a time.
- Set up maintenance. Decide whether the catalog service or your own jobs handle compaction, snapshot expiration and orphan file cleanup.
- Monitor and audit. Centralize access logs and commit history, and alert on failed commits and unusual access patterns.
Moving Tables Between Catalogs Safely
Because an Iceberg table is defined by its metadata files in storage, changing catalogs does not require copying data. It does require care, because for a period two catalogs may know about the same table. A safe migration follows a simple sequence:
- Freeze schema changes on the tables in scope for the duration of the move, and announce the window to owners of every writer.
- Register, do not copy. Register each table in the new catalog using its latest metadata location, then verify snapshot history, schema and partition spec match.
- Switch readers first. Point read-only consumers at the new catalog and confirm results match. Readers can move back instantly if something is wrong.
- Switch writers one at a time. Stop the writer, confirm the new catalog holds the latest snapshot, reconfigure the writer, and resume. Never let both catalogs accept commits for the same table.
- Retire the old entry. Once all writers have moved, drop the old registration without deleting data, or make it read-only, so no one commits through it by accident.
Plan the permission model in the new catalog before switching anyone, and test it with real user identities. A migration that preserves data but loses access rules simply moves the risk somewhere less visible.
A Worked Example: A Multi-Cloud Retailer
Picture a retailer with an AWS-based data platform running Spark on EMR, a Snowflake account used by finance for reporting, and a data science team on Google Cloud. Tables are Iceberg in S3, but each team has registered them differently: some in Glue, some only as external tables in Snowflake, and some read directly by path from notebooks.
The platform team’s goal is one system of record per table and consistent policy. Because no single vendor’s compute dominates and two clouds must have first-class access, they select a Polaris-based catalog as the neutral system of record for shared domain tables. Snowflake connects to it as an external Iceberg catalog for finance reporting; Spark on EMR and notebooks on Google Cloud use the REST protocol with vended credentials. Path-based access is turned off by removing broad storage permissions.
Sensitive customer tables require masking for most users. The team verifies which engines enforce the catalog’s policies and, for paths where enforcement is incomplete, publishes masked derived tables instead of granting access to the raw ones. Within three months, every shared table has one owner, one catalog entry and one permission model, and adding a new engine becomes a configuration task rather than a data copy.
The team also documents what it deliberately did not do. Finance’s Snowflake-native tables, which no other engine needs, stay in Snowflake’s own catalog. Data science feature tables that only exist on Google Cloud stay in Google’s catalog, with a scheduled review if other teams start to depend on them. Neutrality is applied where sharing happens, not imposed everywhere for its own sake, which keeps the operational burden proportional to the benefit.
Pitfalls and Outlook
- Assuming REST compatibility means feature parity. Core operations are standard, but policy enforcement, server-side planning and maintenance vary.
- Leaving path-based access in place. Any engine reading files directly bypasses the catalog’s governance and audit.
- Two catalogs accepting writes for one table. This creates conflicting commits and broken tables.
- Ignoring commercial terms. External access billing and request-based pricing can change the economics of a design.
- Treating the catalog as an afterthought. Retrofitting governance after dozens of teams have built on ad hoc registrations is far harder than designing it first.
Looking ahead, expect catalogs to absorb more responsibilities: server-side scan planning, automatic optimization, governance of AI assets and semantic definitions, and cross-catalog federation. The REST protocol will keep core interoperability intact, while differentiation moves to policy enforcement across engines and to how much operational work each service removes.
Conclusion
The catalog is now the most consequential choice in lakehouse architecture. Unity Catalog is strongest for Delta-heavy and Databricks-centric estates that want to govern AI assets alongside tables. Apache Polaris is the neutral choice for multi-engine, multi-cloud Iceberg. Snowflake Horizon, AWS Glue and Google’s runtime catalog are pragmatic options for organizations centered on those platforms. Decide based on formats, engines, clouds and policy enforcement requirements, insist on one system of record per table, and eliminate path-based access. Get that right and the promise of the open lakehouse, one copy of data and many engines, becomes real.
How Hanoi Technology Can Help
Hanoi Technology designs and implements lakehouse catalogs and governance models across Databricks, Snowflake, AWS and Google Cloud.