An incremental load processes only data that is new or changed since the last run, instead of reloading everything. It is cheaper and faster than a full refresh but needs a reliable way to find what changed, such as a watermark column or a change feed, and care at the boundaries.
For example, an events table in a source database grows by about a million rows a day and holds three years of history. A full reload would read over a billion rows every night. An incremental load stores the highest created_at it has loaded, reads only rows with a later value on the next run, appends them and moves the stored value forward. The nightly job reads one day of rows instead of three years.
Boundaries are where incremental loads go wrong. Rows with the same timestamp as the stored value can be missed if the next run reads strictly greater values, or loaded twice if it reads greater-or-equal values without deduplicating. Rows committed late, after a run has moved past their timestamp, are skipped unless the run re-reads a short overlap window. Updates and deletes are invisible to a pure append, which is why sources whose rows change need change data capture instead.
A periodic full reload, or a check that compares row counts between source and target, catches drift that incremental runs miss. Platforms provide their own tools for incremental work, such as Auto Loader and streaming tables on Databricks, and streams and dynamic tables on Snowflake. See also idempotency and late-arriving records.
Whatever the method, an incremental pipeline should record what each run read, such as the watermark before and after, so a gap can be found and filled later with a backfill.
Emolysis covers incremental loading with two shapes: Incremental append for append-only sources and CDC merge for sources whose rows change. A snapshot is the alternative for small sources. The risk of an incremental load is a silent gap or a duplicate at the boundary, which is why its tests use runs that overlap.