Glossary

Late-arriving records

Also known as: late-arriving facts · late data · late-arriving dimensions

Definition

Late-arriving records are facts that reach the pipeline after the period they belong to was already processed, such as an offline mobile event or a partner file delivered days late. If the pipeline ignores them, totals for past periods stay wrong.

For example, a delivery app records orders on drivers' phones and uploads them when the phone has a connection. An order placed on Monday evening in an area without coverage reaches the pipeline on Wednesday morning. By then, Monday's totals have been computed and published. To be correct, the pipeline must add the order to Monday, recompute Monday's aggregates and publish them again, while leaving Tuesday alone unless something else changed there.

Two timestamps make this work: the event time, when the order happened, and the arrival or load time, when the pipeline received it. Partitioning facts by event time, and tracking which event-time partitions received new rows in each load, tells the pipeline exactly which windows to rebuild. A limit on how late an event may be, such as seven days, keeps the reprocessing bounded; anything later is logged for review.

The dimension side has its own form of the problem: a fact can refer to a customer the pipeline has not seen yet. A common answer is to insert a placeholder dimension row, sometimes called an inferred member, and fill in its attributes when the customer record arrives, so the fact is not dropped and the join still works. See also backfill, partitioning and incremental load.

Platforms help with the mechanics: on Databricks, a merge into a Delta table can update the affected days in place, and Snowflake's streams show which rows arrived since the last run. The decision of how late is too late is a business one, and it should be written down next to the pipeline. See also watermark.

Handling them means reprocessing the affected windows and linking facts whose dimension rows have not arrived yet to a placeholder. Emolysis's Late-arriving shape does both, tested with a fact that arrives two days late. See also fact table and watermark.