For two decades, data integration meant a nightly job: extract yesterday’s rows, transform them in a staging area, and load them into a warehouse before the business woke up. That rhythm is breaking. Operational dashboards are expected to reflect the last few minutes, fraud and inventory decisions happen inside a transaction window, and AI agents that answer customer or employee questions are only as trustworthy as the freshness of the data they retrieve. In 2026 the question most data teams are asking is no longer whether to adopt change data capture (CDC) and streaming, but how to make it the default without creating a fragile tangle of connectors.
This article explains what has shifted in the integration market over the past eighteen months, how a log-based CDC architecture actually works, where it fails in production, and how to decide what should be real-time and what can stay in batch.
What Changed: Three Forces Behind the Shift
CDC is not new. Debezium, the open source CDC platform built on Kafka Connect, has been capturing row-level changes from database logs for years. What changed is the economics and the demand side.
1. Platform vendors made CDC a built-in feature
The major cloud and data platforms now ship managed, log-based replication as a first-class capability rather than an add-on. In November 2025, AWS announced AWS Glue zero-ETL for self-managed databases, continuously replicating Oracle, SQL Server, MySQL and PostgreSQL (on-premises or on EC2) into Amazon Redshift. Databricks made its SQL Server connector in Lakeflow Connect generally available in September 2025, using SQL Server’s CDC and Change Tracking features for incremental ingestion with SCD Type 2 history. Snowflake’s Openflow, built on Apache NiFi flow definitions, lists database CDC connectors for MySQL, Oracle, PostgreSQL, SQL Server and MongoDB alongside SaaS sources such as Salesforce, Workday and Jira Cloud. When the destination platform offers replication out of the box, the cost of choosing streaming over batch drops sharply.
2. The integration market consolidated around “AI-ready” data
In June 2026, Fivetran and dbt Labs completed their all-stock merger, which TechTarget reported was positioned explicitly as a data infrastructure layer for AI agents: Fivetran moves the data, dbt models and documents it. Whatever one thinks of vendor consolidation, the signal is clear. Ingestion is being sold not as plumbing but as the first step in a governed pipeline that ends in a semantic layer that agents can query.
3. Consumers became continuous
Retrieval-augmented generation, operational analytics, personalization engines and event-driven microservices all consume data continuously. A support agent grounded on a customer record that is twenty hours stale will confidently give the wrong answer about an order that shipped this morning. Freshness has moved from a nice-to-have dashboard property to a correctness requirement for downstream automation.
How Log-Based CDC Works
Every serious relational database keeps a transaction log for durability and replication: the write-ahead log (WAL) in PostgreSQL, the binlog in MySQL, redo logs in Oracle, and the transaction log read by SQL Server’s CDC feature. Log-based CDC reads that log, not the tables, and turns each committed insert, update and delete into an ordered event.
That design gives it three advantages over the alternatives:
- Low source impact. Query-based CDC (polling a last_updated column) scans tables repeatedly and misses hard deletes. Trigger-based CDC adds write overhead to every transaction. Reading the log avoids both.
- Completeness. Deletes, intermediate updates and the before-and-after image of a row are all captured, which matters for audit, SCD Type 2 history and reconciliation.
- Ordering and replay. Events carry transaction metadata, and when they are written to a durable log such as Kafka they can be replayed to rebuild a downstream table or bootstrap a new consumer.
A typical pipeline has four stages. An initial snapshot copies the current state of each table. The connector then switches to streaming from the log position recorded at snapshot time. Events land in a transport layer (Kafka, Kinesis, Pub/Sub, or a platform’s internal queue). Finally a sink applies them to the target, typically merging into Delta Lake or Apache Iceberg tables, a warehouse, a search index or a cache.
Where stream processing fits
Raw change events are rarely what consumers want. Stream processors such as Apache Flink (including the Flink CDC project), Kafka Streams, or streaming databases like RisingWave let teams join, filter, deduplicate and aggregate change streams into materialized views before they reach consumers. The rule of thumb: keep the capture layer dumb and faithful, and put business logic in a processing layer that can be versioned, tested and replayed.
Table changes versus business events
One subtle design choice deserves attention early. Row-level change events describe what happened to a table, not what happened in the business. An order that moves to “shipped” might touch four tables in one transaction, and a downstream consumer reconstructing that from raw changes has to understand the application’s internal schema. That couples every consumer to the source database design, which is exactly the coupling microservice teams try to avoid.
The common remedy is the transactional outbox: the application writes a well-defined business event (for example, OrderShipped with a stable payload) to an outbox table in the same transaction as its state change, and CDC captures that table. Consumers then subscribe to meaningful, versioned events while still getting the reliability of log-based capture. A sensible split is to use raw table CDC for analytics replication, where full fidelity matters, and outbox events for operational integration between services and agents, where a stable contract matters more than a mirror of the schema.
The Production Pitfalls Nobody Puts in the Demo
CDC demos look effortless. Production CDC fails in ways that are specific and predictable. Confluent’s guidance on its Debezium-based PostgreSQL connector, for example, warns about long-running transactions delaying change streaming and growing the WAL, and about inactive replication slots retaining WAL files until disks fill. These are the issues we see most often:
- Replication slot and log retention. If a connector stops consuming, the database keeps log files it would otherwise discard. A paused PostgreSQL slot can fill a disk and take down the primary. Monitor slot lag and set alerts well before storage limits.
- Schema evolution. A column rename or type change upstream can break deserialization or silently drop data downstream. Use a schema registry with compatibility rules and treat source schema changes as a contract change, not a surprise.
- Snapshot pain. Initial snapshots of multi-terabyte tables can take days and lock resources. Incremental or chunked snapshotting and running against a replica reduce risk.
- Duplicates and ordering. Most pipelines guarantee at-least-once delivery. Sinks must be idempotent, typically merging on primary key and source log position, so that a replayed event does not create a second row.
- Permissions and security. CDC users need replication privileges that security teams rightly scrutinize. Plan grants, TLS and network paths early, especially for on-premises sources.
- Hard deletes and GDPR erasure. A delete event must propagate to every downstream copy, including Iceberg snapshots and caches, or an erasure request is only partly honored.
CDC is easy to start and hard to operate. The teams that succeed treat every connector as a production service with an owner, a lag SLO, schema contracts and a tested replay procedure.
Deciding What Should Be Real-Time
Not everything deserves a stream. Streaming infrastructure costs money and on-call attention, and many analytical workloads are perfectly served by hourly or daily loads. A simple way to decide is to score each data flow against four questions.
- Decision latency: How quickly does someone or something act on this data? If the answer is “next morning,” batch is fine.
- Cost of staleness: What goes wrong if the data is a day old? Wrong answers from an agent, oversold inventory or missed fraud justify streaming; a slightly stale monthly KPI does not.
- Change volume and pattern: High-churn tables are often cheaper to replicate incrementally than to reload fully, even when consumers only read them daily.
- Source fragility: Heavy nightly extracts against an OLTP database can be riskier than a steady, low-impact log reader.
In practice, most enterprises end up with a tiered model: a small set of operational and AI-facing domains replicated continuously with second-to-minute latency, a larger set of tables replicated incrementally but merged in micro-batches every few minutes to an hour, and a long tail of reference and archival data loaded in batch.
A Reference Architecture for 2026
The architecture we recommend most often has five layers, each replaceable without rebuilding the others.
- Capture: managed platform connectors where they exist and meet requirements; Debezium or Flink CDC where you need portability, unusual sources, or control over event formats.
- Transport: a durable, replayable log (Kafka or a managed equivalent) when more than one consumer needs the same changes; direct platform ingestion when there is only one destination.
- Processing: Flink, Spark Structured Streaming or a streaming database for joins, deduplication and enrichment, with code in version control and tests in CI.
- Storage: open table formats (Iceberg or Delta) as the system of record for analytics, so multiple engines can read the same data.
- Serving: semantic models, APIs, search indexes and vector stores fed from curated tables, each with a declared freshness target.
Two cross-cutting concerns make or break this architecture: observability (end-to-end lag, throughput, error rates and data quality checks per pipeline) and lineage, so that when a dashboard or agent answer is wrong you can trace it back to the connector and the source transaction that produced it.
Worked Example: An Order Platform Feeding Analytics and an AI Assistant
Consider a mid-size retailer with an order management system on PostgreSQL, an ERP on SQL Server, and a CRM in a SaaS application. The business wants a near-real-time operations dashboard and a customer-service assistant that answers “where is my order?” questions accurately.
A pragmatic design would look like this. Orders and shipments in PostgreSQL are captured with Debezium into Kafka, because three consumers need them: the lakehouse, a search index used by the assistant, and a fraud service. ERP inventory tables are ingested through the lakehouse platform’s managed SQL Server connector, since the lakehouse is the only consumer and the managed option removes connector operations. CRM data arrives through a managed SaaS connector on a 15-minute schedule, which is fresh enough for account context.
In the processing layer, a Flink job joins order and shipment events into an order_status stream and writes it both to an Iceberg table and to the search index. The assistant’s retrieval tool reads only the curated order_status view, never raw tables, and each answer cites the record’s last-updated timestamp. The team sets explicit SLOs: under one minute end-to-end lag for order status, under 30 minutes for inventory, and daily for historical analytics. Alerts fire on replication slot lag and on schema changes detected by the registry.
The result is not the most elegant architecture on paper, mixing open source and managed components, but each choice is justified by consumers, latency and operational cost. That is what good integration design looks like in 2026.
Implementation Checklist
- Inventory sources and consumers, and record a freshness requirement for each data flow before choosing tools.
- Confirm database prerequisites: log retention, replication settings, privileges, and whether you can read from a replica.
- Choose managed connectors by default and justify each self-managed one with a concrete need.
- Adopt a schema registry and compatibility policy; route breaking changes through a review.
- Make every sink idempotent and test a full replay from a known offset before go-live.
- Define lag SLOs per pipeline and alert on them, not only on connector status.
- Propagate deletes end to end and document how erasure requests reach every copy.
- Emit lineage from capture through serving so incidents can be traced quickly.
Outlook: Integration as a Product, Not a Pipeline
Over the next two years expect three developments. First, the boundary between operational databases and analytics will keep blurring as platforms push replication deeper into their engines, making “zero-ETL” the default for simple, single-destination cases. Second, CDC streams will increasingly be governed like data products, with contracts that define schema, semantics and freshness, because downstream AI systems need guarantees rather than best effort. Third, agents themselves will become integration consumers and producers, reading fresh state and writing back actions, which raises the bar for idempotency, auditability and access control at the integration layer.
Conclusion
Real-time data integration is no longer an exotic capability reserved for tech giants. Managed CDC from every major platform, mature open source tools and continuous consumers such as AI assistants have made it the sensible default for data that drives decisions. The hard part has moved from “can we capture changes?” to “can we operate dozens of change streams reliably, with contracts, observability and lineage?” Teams that answer that question deliberately, choosing streaming where staleness is costly and batch where it is not, will have data foundations that analytics and AI can actually trust.
How Hanoi Technology Can Help
Hanoi Technology designs and operates CDC and streaming pipelines, from connector selection to lag SLOs and lineage, as projects, dedicated teams or managed services.
Further Reading
- Announcing AWS Glue zero-ETL for self-managed database sources – AWS
- SQL Server connector in Lakeflow Connect now generally available – Databricks
- Openflow connectors – Snowflake Documentation
- CDC and data streaming with Debezium – Confluent
- Fivetran, dbt Labs complete merger to form data layer for AI – TechTarget
- CDC Stream Processing: The Complete Guide – RisingWave