For years, real-time data and analytical data lived in different worlds. Event streams flowed through Kafka to power applications, while the lakehouse received batch extracts hours later. Bridging the two meant custom consumers, Spark jobs writing tiny files, and nightly compaction scripts that someone had to babysit. The result was stale dashboards, duplicate pipelines and two versions of the truth.
In 2025 and 2026 that gap has narrowed sharply. Streaming vendors now write Kafka topics directly into Apache Iceberg tables, open-source projects such as Apache Fluss unify streaming storage with lakehouse tiering, and Iceberg v3 added row lineage and deletion vectors that make continuously updated tables more practical. This article explains the main patterns for building a streaming lakehouse, the tradeoffs among them, and a playbook for implementing one that stays fast and maintainable.
Why Streams and Tables Are Converging
Three pressures are driving the convergence. First, AI and operational analytics need fresh data: fraud models, inventory agents and customer-facing recommendations lose value with every hour of delay. Second, maintaining separate streaming and batch copies of the same data is expensive and error-prone. Third, open table formats have become the shared language of analytics engines, so landing streaming data in Iceberg makes it immediately available to Spark, Trino, Snowflake, Databricks, DuckDB and BigQuery without further copying.
The core technical challenge is that streams and tables have different shapes. Kafka is an append-only log optimized for many small writes and sequential reads. Iceberg is optimized for large, columnar files with metadata that engines can prune efficiently. Writing every event as it arrives produces thousands of tiny files and an explosion of snapshots. Writing too rarely makes data stale. Every streaming lakehouse design is a way of managing that tension.
The Main Patterns for Kafka to Iceberg
1. Managed topic-to-table services
Confluent Tableflow, generally available since March 2025, materializes Kafka topics as Iceberg tables, with Delta Lake support added in preview. It uses Schema Registry as the source of truth, converts Avro, JSON or Protobuf to Parquet, handles schema evolution and type mapping, compacts small files and expires old snapshots. Tables can be exposed through an embedded Iceberg REST catalog or registered with external catalogs such as AWS Glue and Apache Polaris. Transformations, filtering and deduplication before landing require a separate stream processing step. WarpStream offers a similar capability in a bring-your-own-cloud model.
2. Broker-native Iceberg topics
Some Kafka-compatible platforms write Iceberg directly from the broker. Redpanda made Iceberg Topics generally available in its 25.1 release, and Aiven has introduced Iceberg topics for Apache Kafka built on tiered storage. These approaches minimize moving parts, but capabilities vary: some handle snapshot expiration but leave compaction to you, some cannot backfill existing topics, and some prioritize zero-copy storage over freshness. Read the documentation for maintenance and backfill behavior carefully.
3. Kafka Connect Iceberg sink
The open-source Iceberg sink connector for Kafka Connect is mature and flexible, with support for multiple tables, schema evolution and exactly-once commit coordination. It suits organizations already running Kafka Connect at scale. Its main cost is operational: you own the Connect cluster, commit intervals, and all table maintenance such as compaction and snapshot expiration.
4. Stream processing engines
Apache Flink and Spark Structured Streaming can read from Kafka, transform, join and aggregate, and then write to Iceberg. This is the right choice when data needs real processing before it lands: deduplication, enrichment from reference data, windowed aggregations or upserts keyed by primary key. It is also the most engineering-intensive option.
5. Streaming storage with lakehouse tiering
Apache Fluss, which graduated to a top-level Apache project in August 2026, takes a different approach. It is a columnar streaming storage layer built on Apache Arrow, designed for sub-second freshness, primary-key tables with updates and lookups, and first-class Flink integration. Recent data stays in Fluss while history tiers into Iceberg or Apache Paimon, and a union read presents both as one table. It suits use cases that need very fresh, updatable data alongside long-term lakehouse history.
Freshness is easy; maintained freshness is hard. The pattern you choose should be judged by who compacts, who expires snapshots and who handles schema change, not just by how fast the first event lands.
Decision Criteria
Use the following criteria to narrow the choice for each stream, rather than forcing one pattern on every topic:
- Freshness requirement. Minutes are achievable with most patterns; seconds require stream processing or streaming storage such as Fluss; hours or days suggest batch is still fine.
- Transformation needs. Raw event archiving fits topic-to-table services; deduplication, joins and upserts need a stream processor.
- Update semantics. Append-only events are simple. CDC streams with updates and deletes need upsert support and benefit from Iceberg v3 deletion vectors.
- Maintenance ownership. Managed services handle compaction and expiration; self-run connectors and processors leave it to you.
- Catalog integration. Tables should register in your lakehouse’s system-of-record catalog, not only in an embedded one.
- Ecosystem and cost. Managed services trade higher unit prices for lower operational effort; open-source paths trade engineering time for flexibility.
- Backfill. Check whether the approach can materialize history already in the topic or only new data.
Applying the criteria to typical streams
In practice, three stream archetypes cover most enterprise topics. Clickstream and application logs are high volume, append-only and tolerant of a few minutes of delay; a managed topic-to-table service or the Connect sink is usually enough, with VARIANT columns for flexible payloads. Database change streams from operational systems carry updates and deletes and must preserve ordering per key; they need upsert-capable processing and tables with efficient delete handling. Business events such as orders, payments and shipments are moderate in volume but high in value, often need enrichment and deduplication, and feed both real-time decisions and financial reporting; these justify a stream processor and careful data contracts. Classifying each topic into one of these archetypes quickly narrows the technology choice and avoids building a heavyweight pipeline for data that only needs to be archived.
Reference Architecture
A pragmatic streaming lakehouse usually combines patterns in layers:
- Raw landing. Topics with registered schemas land as append-only Iceberg tables through a managed topic-to-table service or the Connect sink. This gives a replayable, queryable history of every event.
- Refined streaming tables. Flink or Spark jobs read high-value topics, deduplicate, enrich and upsert into curated Iceberg tables keyed by business identifiers.
- Low-latency serving. Where sub-second access or key lookups are needed, a streaming storage layer or operational database serves the latest state, tiering history to the lakehouse.
- Batch and AI consumption. Warehouses, BI tools, feature pipelines and notebooks query the same Iceberg tables through the shared catalog.
- Maintenance and observability. Compaction, snapshot expiration, orphan file cleanup, freshness monitoring and data quality checks run continuously.
Implementation Playbook
Step 1: Classify streams by value and freshness
List your topics with their owners, volume, schema quality and the freshness that consumers actually need. Many teams discover that only a handful of streams need minute-level freshness; the rest can land on a slower, cheaper cadence.
Step 2: Fix schemas at the source
Streaming into tables exposes poor schemas immediately. Require registered schemas with compatibility rules for every topic that lands in the lakehouse, and agree with producing teams on how breaking changes are handled. A data contract between producers and the platform team prevents silent failures downstream.
Step 3: Choose commit intervals deliberately
Commit frequency controls the balance between freshness and file count. Short intervals create many small files and snapshots; long intervals delay data. Start with the slowest interval that meets consumer needs, measure file sizes and snapshot counts, and tighten only where justified.
Step 4: Automate table maintenance
Schedule compaction for streaming tables, expire snapshots according to your time-travel and audit requirements, and remove orphan files. If you use a managed service, document precisely which of these it performs. Monitor average file size and snapshot count per table as health indicators.
Step 5: Handle updates and deletes properly
For CDC streams, use a processor or service that supports upserts by primary key. Prefer Iceberg v3 tables with deletion vectors where all engines support them, since they make frequent row-level changes far cheaper than older delete mechanisms. Test how late and out-of-order events are handled.
Step 6: Monitor freshness and quality end to end
Track the lag between event time in Kafka and availability in the table, not just connector health. Add quality checks on volume, nulls and key uniqueness, and alert owners when a stream deviates from its baseline.
Step 7: Plan for replay and recovery
Things will go wrong: a bad deployment writes malformed records, a schema change slips through, or a processor falls behind for hours. Decide in advance how you will recover. Keep Kafka retention long enough to replay from the last known good point, use Iceberg snapshots to roll a table back to a clean state, and document the procedure so on-call engineers are not improvising at night. Test the recovery runbook at least once before production traffic depends on it.
A Worked Example: Real-Time Inventory for an Omnichannel Retailer
Consider a retailer streaming point-of-sale transactions, e-commerce orders and warehouse stock movements through Kafka. Merchandising wants inventory positions refreshed within minutes, while finance needs a complete, auditable history.
The platform team lands all three topics as raw append-only Iceberg tables using a managed topic-to-table service, registered in the company’s central catalog. This immediately gives analysts queryable history without custom code. For inventory, a Flink job joins sales and stock movements, deduplicates retried events and upserts current stock per product and location into a curated Iceberg v3 table, with a commit interval of a few minutes agreed with merchandising.
Compaction runs hourly on the curated table and daily on raw tables, and snapshots older than the audit window are expired. A freshness monitor alerts if inventory data is more than fifteen minutes behind event time. Finance continues to run month-end reports from the raw tables, now with no separate batch extract. The organization ends up with one pipeline per stream instead of a streaming path and a batch path, and every consumer reads from the same governed tables.
Cost and Capacity Planning
Streaming lakehouse costs hide in places batch teams rarely look. Object storage charges for requests as well as capacity, so frequent small commits and metadata reads can matter as much as data volume. Compaction consumes compute continuously. Managed topic-to-table services usually charge by data processed or by table, while self-run connectors and processors cost cluster time plus engineering effort. Cross-zone and cross-region network transfer can be significant if brokers, processors and storage are not co-located.
Build a simple model per stream before choosing a pattern: events per second, average event size, commit interval, expected file sizes after compaction, retention, and the number of engines querying the table. Then compare at least two options using vendor price sheets and a short load test with production-like traffic. Revisit the model quarterly, because stream volumes tend to grow faster than anyone forecasts, and a pattern that was cheapest at launch may not stay that way.
Pitfalls to Avoid
- Small-file explosions. Aggressive commit intervals without compaction degrade query performance and raise storage request costs.
- Schema drift from producers. Unregistered or incompatible schema changes break tables or silently drop fields.
- Duplicate events. At-least-once delivery means duplicates; deduplicate where downstream metrics require exactness.
- Embedded catalogs as islands. Tables visible only through a vendor’s embedded catalog fragment governance. Register in your system-of-record catalog.
- Ignoring backfill. Some approaches only capture new data; plan how history will be loaded.
- Measuring connector uptime instead of freshness. A healthy connector can still deliver stale tables if commits or maintenance lag.
Outlook and Conclusion
The boundary between streaming platforms and lakehouses will keep blurring. Expect more streaming services to support upserts, dead-letter handling and custom partitioning natively, more lakehouse engines to read the latest streaming data through union reads, and Iceberg row lineage to make incremental processing downstream far simpler. Projects like Fluss suggest a future where “streaming table” and “lakehouse table” are two tiers of the same object.
For now, the practical path is clear. Classify streams by the freshness they truly need, land raw events in governed Iceberg tables with a managed or connector-based approach, use stream processing where data needs real transformation or upserts, and treat table maintenance and freshness monitoring as first-class parts of the platform. Done well, a streaming lakehouse removes the old split between real-time and analytical data and gives every consumer, human or AI, the same fresh, trustworthy tables.
How Hanoi Technology Can Help
Hanoi Technology builds streaming lakehouse pipelines from Kafka to governed Iceberg tables, including maintenance, monitoring and data contracts.