# Backfill

> A backfill re-runs a pipeline for a past period, to load history, apply a fix to old data or reprocess data a source delivered again.

A backfill re-runs a pipeline for a past period, to load history for a new table, apply a fix to old data, or reprocess data a source delivered again. A safe backfill replaces exactly the [partitions](https://emolysis.com/glossary/partitioning) in its range, runs in order, can resume after a failure and gives the same result if run twice.

For example, suppose a bug in a currency conversion wrote wrong amounts into a daily revenue table from March 1 to March 14. After the fix, a backfill re-runs the pipeline for those fourteen days, one day at a time. Each run reads the source data for its day, rebuilds that day's partition and replaces it in a single write, so readers see either the old day or the new one, never half of each. Days outside the range keep their data.

Order and size matter. Running the oldest day first keeps running totals and snapshots consistent, and small steps mean a failure on day nine leaves days one to eight finished, so the backfill can resume from day nine instead of starting over. Each step should be [idempotent](https://emolysis.com/glossary/idempotency): if day nine runs twice, the table looks the same as if it ran once. Backfills over long ranges also need a limit on how many days run in parallel, so they do not crowd out the daily jobs or overload a source system.

A backfill writes over history, so it deserves the same review as a code change: which range, which tables, and which reports it changes. Tables [partitioned](https://emolysis.com/glossary/partitioning) by date make it cheaper, because a day can be replaced without rewriting the table. See also [incremental load](https://emolysis.com/glossary/incremental-load) and [watermark](https://emolysis.com/glossary/watermark), which decide what a normal run reads.

[Backfill](https://emolysis.com/shapes/backfill) is one of Emolysis's ten pipeline shapes, with a golden test that checks a gap in the middle of a range is filled without touching the rest. It differs from [late-arriving records](https://emolysis.com/glossary/late-arriving-data), which reprocess only the windows that new facts touched.

---

Page: https://emolysis.com/glossary/backfill. Emolysis is a product of Bandpey GmbH, Munich, Germany. It is in private development, and nothing is released yet.