Databricks
The Databricks template runs MERGE into Delta fact tables, then uses replaceWhere on the affected partitions of the aggregates.
- MERGE lands the new facts by key without duplicates. The batch's event dates then tell the template which partitions of the aggregate tables to recompute with replaceWhere.
- Streaming pipelines need a lateness allowance. A streaming watermark drops events later than its allowance in stateful operations such as windowed aggregates, and a batch correction like this one is the usual way to catch what it dropped.
- A placeholder member for early-arriving facts is an ordinary row in the dimension table with a reserved key.
- Facts join to the dimension version that was valid at their event time, which is why this shape pairs with SCD Type 2.