Glossary

Change data capture (CDC)

Also known as: CDC · change feed · change data feed

Definition

Change data capture records the inserts, updates and deletes made to a source table, usually from the database's transaction log, so downstream systems can apply only what changed instead of copying the whole table. A change feed carries each change with its key, the operation and an ordering value such as a commit timestamp.

For example, a customers table in an operational database changes all day: new customers sign up, addresses change and closed accounts are deleted. Instead of copying the whole table every night, CDC delivers a stream of changes such as customer 42 updated with a new address at 09:14, customer 77 inserted at 09:20 and customer 13 deleted at 10:02. The pipeline applies those changes in order to its copy of the table, and only the rows that changed are read and written.

Most CDC tools read the database's own log of committed transactions, so they see every change in commit order without querying the table. SQL Server has built-in change data capture and change tracking, and PostgreSQL offers logical replication. Other sources only offer an updated-at column, which misses deletes and can miss updates that share a timestamp, so log-based CDC is preferred where the source supports it.

The hard cases are ordering and repeats. Several changes to the same key can arrive in one batch, so the pipeline must apply only the last one, ranked by the ordering value; a batch can be delivered twice, so applying it must be idempotent; and deletes need their own branch in the merge. See incremental load, watermark and the SQL Server and PostgreSQL source pages.

On the target side, Databricks can expose the changes made to a Delta table through Change Data Feed, and Snowflake streams track changes to a table, so CDC can also feed pipelines inside the platform. See also merge.

Platforms have their own forms, such as Delta Change Data Feed and Snowflake streams. Emolysis's CDC merge shape applies a change feed to a target table with a merge. PostgreSQL is in development as a source and SQL Server is planned as one; see Sources.