One of the biggest promises of the open lakehouse is that storage and compute are decoupled. Store your data once in an open table format such as Apache Iceberg, govern it through a shared catalog, and let different engines read and write it for different jobs. In 2026 that promise is largely real: Spark, Trino, Flink, StarRocks, ClickHouse, DuckDB and the major cloud warehouses all read Iceberg tables to varying degrees, and several write them.
Choice, however, creates its own problem. Teams now face a crowded field of engines with overlapping features, enthusiastic vendor benchmarks and very different operational profiles. This guide is for data platform leads and architects deciding which engines to run on their lakehouse. It explains how engines differ, maps them to workloads, proposes evaluation criteria and a testing approach, and warns about common mistakes.
Why Engine Choice Matters More Now
In a traditional data warehouse, choosing the platform meant choosing the engine. In a lakehouse, you can, and often should, run more than one. Three developments make this decision more consequential than it was a few years ago.
- Open formats matured. Iceberg has become a common denominator across vendors, with REST catalogs enabling engines from different providers to share tables.
- Workloads diversified. The same data now serves batch transformation, streaming, interactive BI, customer-facing analytics, data science and AI agents, each with different latency and concurrency needs.
- Engines specialized. Some engines optimize for large-scale transformation, others for federated SQL, others for sub-second dashboards at high concurrency. No single engine is best at everything.
In the open lakehouse, the question is not which engine wins. It is which small set of engines covers your workloads with the least operational overhead.
The Main Engine Families
Apache Spark: the general-purpose workhorse
Spark remains the most widely used engine for large-scale data transformation, machine learning feature pipelines and batch processing on lakehouse tables. Spark 4.1, announced in December 2025, added Spark Declarative Pipelines, which let teams define datasets and queries in Python or SQL while Spark manages dependencies, retries and checkpoints. It also introduced a Real-Time Mode for Structured Streaming targeting sub-second latency for certain stateless queries, made SQL scripting generally available, and brought VARIANT support for semi-structured data to general availability, according to Databricks’ announcement.
- Strengths: broad language support, mature Iceberg read and write support, rich ecosystem, strong for heavy transformations, table maintenance and ML.
- Limitations: not designed for very low-latency, high-concurrency interactive queries; cluster startup and tuning add overhead.
Trino: federated, interactive SQL
Trino is a distributed SQL engine known for interactive analytics and federation across many sources. It can join Iceberg tables with data in relational databases, other lakes and SaaS systems through a large connector ecosystem, and it supports writing to Iceberg.
- Strengths: federation, ANSI SQL, broad connector coverage, strong community and several commercial distributions.
- Limitations: historically weaker at very high concurrency and sub-second latency without caching; coordinator design affects high availability planning. Recent releases have added file system caching options, so evaluate current versions rather than older comparisons.
StarRocks and similar real-time OLAP engines
StarRocks is a native, vectorized MPP engine written in C++ that can query Iceberg and other lake formats directly, with caching and materialized views designed for low-latency analytics. Comparable engines include Apache Doris and, for some use cases, ClickHouse with its lakehouse integrations.
- Strengths: low latency and high concurrency for dashboards and customer-facing analytics; data caching; automatic query rewrite to materialized views.
- Limitations: narrower federation than Trino; write support to lake formats is more limited; operational model differs from Spark-centric teams.
A public case study from TRM Labs, published in January 2025, illustrates the trade-off. Evaluating engines for a multi-terabyte Iceberg workload with strict latency targets at high concurrency, the team found StarRocks outperformed Trino in their tests, particularly under concurrent load, and chose Iceberg with StarRocks. They also tested DuckDB, which performed respectably on a single node but was limited by Iceberg predicate pushdown at the time. The lesson is not that one engine is universally faster, but that concurrency and caching behavior can dominate results for serving workloads.
Single-node and embedded engines
DuckDB, Polars and similar engines run in a single process and increasingly read Iceberg tables. They are excellent for data science, local development, small to medium analytics and embedded use cases, and can dramatically reduce cost for workloads that do not need a cluster.
Cloud warehouse engines reading open tables
Snowflake, BigQuery, Amazon Redshift, Microsoft Fabric and Databricks SQL can all query Iceberg tables in some form, often with managed performance features and integrated governance. For organizations already invested in one of these platforms, using its engine on open tables can be the simplest route to interactive SQL without operating another cluster.
Streaming engines
Apache Flink and similar engines handle continuous ingestion and stateful stream processing into lakehouse tables. They are usually part of the write path rather than the query layer, but their choice affects table layout and maintenance needs.
Mapping Engines to Workloads
- Large batch transformation and feature engineering: Spark, or a cloud warehouse engine if your transformation stack is SQL-first.
- Streaming ingestion: Flink or Spark Structured Streaming, writing to Iceberg with appropriate commit intervals.
- Ad hoc and federated analytics: Trino or a managed Trino service, especially when queries span the lake and operational databases.
- High-concurrency dashboards and embedded analytics: StarRocks, Doris, ClickHouse or a warehouse engine with result caching.
- Data science and local exploration: DuckDB or Polars, reading Iceberg directly.
- AI agents and natural-language queries: an engine with predictable latency and fine-grained access control, often fronted by a semantic layer.
Evaluation Criteria
- Format and catalog support. Which Iceberg features does the engine read and write (including row-level deletes and newer spec versions)? Does it work with your catalog, such as an Iceberg REST catalog, Polaris, Unity Catalog or a cloud catalog?
- Performance on your workloads. Latency at realistic concurrency, not single-query benchmarks. Include cold and warm cache scenarios.
- Governance integration. Can the engine enforce the catalog’s access policies, row filters and column masks, or will you duplicate security rules?
- Operational model. Managed service or self-run? Kubernetes support, autoscaling, upgrade process, high availability and observability.
- Cost model. Compute pricing, caching storage, data transfer and the staff time needed to operate the engine.
- Ecosystem and skills. Connectors, BI tool compatibility, community activity, commercial support options and whether your team already knows it.
- Write behavior. If the engine writes to shared tables, does it produce well-sized files and respect sort orders and partitioning, or will it create maintenance work?
How to Run a Fair Evaluation
Vendor benchmarks are useful for understanding architecture but rarely predict your results. Many are run by the vendor on a single dataset; the StarRocks project’s own Trino comparison, for example, is candid that it reflects its team’s tests at a point in time. Run your own evaluation.
- Pick representative workloads. Choose ten to twenty real queries per workload class: lookups, aggregations, joins and the slowest queries users complain about.
- Use production-scale data. Copy or snapshot real Iceberg tables, including their actual file layout, rather than freshly compacted test data.
- Test concurrency. Use a load tool to simulate realistic numbers of concurrent users or API calls, and measure p95 and p99 latency.
- Control for caching. Measure cold and warm runs separately, and understand how each engine’s cache behaves after data changes.
- Measure cost. Record compute used per query or per hour of load, including cache storage.
- Assess operations. Perform an upgrade, a node failure and a scale-out during the test. Note how much effort each takes.
- Check governance. Verify that access policies from your catalog are enforced consistently.
Designing a Multi-Engine Lakehouse
Most mature platforms settle on two to four engines. A typical combination is Spark for transformation and maintenance, one interactive SQL engine for analysts, an optional low-latency engine for serving, and DuckDB for data science. To keep this manageable:
- Centralize the catalog. All engines should use the same catalog for discovery, access control and credential vending.
- Assign write ownership. Decide which engine writes each table, and limit writers to avoid conflicting layouts and commit contention.
- Standardize table defaults. Set file sizes, partitioning and sort orders at the table level so all writers behave consistently.
- Route workloads deliberately. Use a semantic layer, query gateway or clear conventions so users know which engine to use for what.
- Monitor centrally. Collect query metrics from every engine into one observability view to spot cost and performance issues.
Decision Checklist
Before adding or replacing an engine, work through the following questions with architecture, security, finance and the teams who will use it. If you cannot answer most of them confidently, run a structured evaluation first.
- Which specific workload or user group is underserved today, and what latency, concurrency and freshness does it need?
- Can an engine you already run meet that need with configuration, caching, materialized views or better table maintenance?
- Does the candidate engine support the Iceberg features and catalog you use, for both reads and, if needed, writes?
- Will it enforce the same access policies as your other engines, without duplicating rules in a separate system?
- Who will operate it, upgrade it and respond to incidents, and do they have the skills today?
- What is the expected monthly cost at realistic load, including caching, storage and staff time?
- How easily could you remove it later if requirements change, and what would depend on it?
- Is there a managed service option, and how does its cost and control compare with running it yourself?
Writing down the answers creates an architecture decision record that future teams can revisit when workloads, engines or prices change. Revisit engine choices at least annually; the lakehouse engine market moves quickly, and a gap that justified a specialized engine one year may be closed by your primary engine the next.
Common Pitfalls
- Choosing by benchmark headline. A benchmark on a different dataset, file layout or concurrency level tells you little about your workload.
- Too many engines. Each engine adds upgrades, security reviews, skills and on-call burden. Add engines only for clear workload gaps.
- Inconsistent security. If engines enforce access differently, sensitive data can leak through the weakest one.
- Ignoring table health. Poorly maintained tables make every engine slow. Fix compaction and snapshot hygiene before blaming the engine.
- Underestimating federation costs. Federated queries against operational databases can overload them. Use federation for exploration and move heavy, recurring queries into the lakehouse.
A Hypothetical Worked Example
A hypothetical e-commerce company stores clickstream, orders and product data in Iceberg on object storage with a REST catalog. It runs Spark for nightly transformations. Analysts complain about slow ad hoc queries, and a new requirement calls for customer-facing analytics in its seller portal, with thousands of concurrent sellers.
- The team keeps Spark for transformation and table maintenance, now using declarative pipelines for new workflows.
- For analysts, it evaluates Trino and its existing cloud warehouse’s engine on Iceberg. The warehouse engine wins on ease of operation and governance integration for this team.
- For the seller portal, it tests StarRocks and the warehouse engine under simulated concurrency. StarRocks, with materialized views over pre-aggregated tables, meets the latency target at lower cost in the team’s tests, so it is adopted for this workload only.
- Data scientists use DuckDB against the same tables for exploration.
- All engines share the catalog, and Spark is the only writer to core tables.
The example is illustrative; a different team with different skills and workloads could reasonably choose differently. The point is that each engine earns its place against a specific workload.
Outlook
Engines are converging on open formats and REST catalogs, which makes switching or adding engines easier than ever. Expect continued investment in caching, vectorized execution and native lakehouse writes across engines, wider support for newer Iceberg features, and tighter integration with semantic layers and AI agents that generate SQL. As AI agents become significant query consumers, predictable latency, cost controls and consistent governance across engines will matter as much as raw speed.
Conclusion
The open lakehouse frees you from a single engine, but it does not free you from choosing well. Map engines to workloads, evaluate on your own data and concurrency, keep the number of engines small, centralize catalog and governance, and assign clear write ownership. Done this way, a multi-engine lakehouse gives each workload the right tool without multiplying cost and complexity.
How Hanoi Technology Can Help
Hanoi Technology helps data teams evaluate and run lakehouse query engines, from benchmarking on real workloads to operating multi-engine platforms on open table formats.