Change Data Capture (CDC)

CDC is a data integration pattern that captures row-level insert, update, and delete events from a source database and streams them to downstream consumers as an ordered event log. It matters because it decouples analytical, caching, and event-driven systems from the operational database without imposing dual-write risk or polling overhead.

Practical example

An e-commerce platform uses Debezium to tail PostgreSQL's logical replication slot for the orders table, streaming change events into Kafka, where a consumer updates an Elasticsearch index and another invalidates a Redis cache entry keyed by order ID — all without the order service ever knowing these downstream systems exist.

CDC implementations fall into two broad categories: log-based and query-based. Log-based CDC (e.g., Debezium reading MySQL binlog, PostgreSQL logical replication slots, or Oracle redo logs) taps the database’s own Write-Ahead Log stream to reconstruct committed transactions in order, without touching application query paths. Query-based CDC instead polls tables using a watermark column (timestamp or auto-increment ID), which is simpler but introduces polling lag, missed deletes, and load on the source. Log-based approaches are strongly preferred at scale because they capture every mutation exactly once per WAL entry and preserve transaction boundaries, but they require the connector to track and persist a durable cursor (LSN, GTID, or binlog file+offset) so it can resume after a crash without gaps or duplication.

The hard architectural problem is not extraction but delivery semantics. Most CDC pipelines guarantee at-least-once delivery to a broker like Kafka, meaning downstream consumers must be idempotent — typically via an Idempotency Key derived from the source LSN or a composite of table/PK/transaction-id. Schema evolution is another persistent edge case: DDL changes (column adds, type widenings, renames) must be captured and versioned alongside data events, or consumers silently deserialize garbage. Tools like Debezium solve this by emitting schema history topics and integrating with a schema registry, but renames and type narrowing still frequently require manual connector intervention or pipeline pause.

Ordering guarantees are scoped per source partition, not globally. If a CDC connector fans out a single table’s changes across multiple Kafka partitions (e.g., keyed by primary key for parallelism), transactions spanning multiple rows can be split across partitions and consumers must reassemble them using the embedded transaction metadata (Debezium’s transaction block) if strict transactional consistency downstream is required. Snapshot bootstrapping is another operational hazard: initializing a CDC pipeline against an existing multi-terabyte table requires a consistent initial snapshot (via a repeatable-read transaction or exported snapshot) stitched to the point in the log where incremental capture begins — get this boundary wrong and you either duplicate or silently drop rows written during the snapshot window.

Failure modes compound under load: a stalled consumer causes the connector’s WAL retention requirement to grow unbounded, which on PostgreSQL can exhaust disk via retained WAL segments behind an unconsumed replication slot, and on MySQL risks binlog purge outrunning the connector’s read position. Production CDC deployments therefore require dedicated monitoring on replication lag, slot/segment retention, and connector offset commit latency as first-class SLOs, not incidental metrics.

CDC is the load-bearing primitive behind cache invalidation, search index synchronization, materialized view maintenance, and cross-region data replication in modern architectures, but it shifts significant operational burden onto schema governance and idempotent consumer design — treating it as a plug-and-play integration tool rather than a distributed streaming subsystem is the most common source of production incidents.