# Incremental load

> An incremental load processes only data that is new or changed since the last run, instead of reloading the whole source every time.

An incremental load processes only data that is new or changed since the last run, instead of reloading everything. It is cheaper and faster than a full refresh but needs a reliable way to find what changed, such as a [watermark](https://emolysis.com/glossary/watermark) column or a change feed, and care at the boundaries.

For example, an events table in a source database grows by about a million rows a day and holds three years of history. A full reload would read over a billion rows every night. An incremental load stores the highest `created_at` it has loaded, reads only rows with a later value on the next run, appends them and moves the stored value forward. The nightly job reads one day of rows instead of three years.

Boundaries are where incremental loads go wrong. Rows with the same timestamp as the stored value can be missed if the next run reads strictly greater values, or loaded twice if it reads greater-or-equal values without deduplicating. Rows committed late, after a run has moved past their timestamp, are skipped unless the run re-reads a short overlap window. Updates and deletes are invisible to a pure append, which is why sources whose rows change need [change data capture](https://emolysis.com/glossary/change-data-capture) instead.

A periodic full reload, or a check that compares row counts between source and target, catches drift that incremental runs miss. Platforms provide their own tools for incremental work, such as Auto Loader and streaming tables on Databricks, and streams and dynamic tables on Snowflake. See also [idempotency](https://emolysis.com/glossary/idempotency) and [late-arriving records](https://emolysis.com/glossary/late-arriving-data).

Whatever the method, an incremental pipeline should record what each run read, such as the watermark before and after, so a gap can be found and filled later with a [backfill](https://emolysis.com/glossary/backfill).

Emolysis covers incremental loading with two shapes: [Incremental append](https://emolysis.com/shapes/incremental-append) for append-only sources and [CDC merge](https://emolysis.com/shapes/cdc-merge) for sources whose rows change. A [snapshot](https://emolysis.com/glossary/snapshot) is the alternative for small sources. The risk of an incremental load is a silent gap or a duplicate at the boundary, which is why its tests use runs that overlap.

---

Page: https://emolysis.com/glossary/incremental-load. Emolysis is a product of Bandpey GmbH, Munich, Germany. It is in private development, and nothing is released yet.