Templates in development

Written for Databricks, Snowflake and Fabric and tested on DuckDB. None has run on a real platform yet, and none is released.

Late-arriving

Late-arriving: correct the right windows.

When facts reach the pipeline after the period they belong to, reprocess only the affected windows downstream. Facts that arrive before their dimension rows are linked to a placeholder and corrected later.

Template in development
Repair
Databricks first

Shape card

07 / 10

  1. Source
  2. Bronze
  3. Silver (used)
  4. Gold (used)
  5. Apps
Group
Repair
Works across
Silver to Gold
Reads
Facts after their period
Writes
Corrected windows
Slots
6 typed slots
Tested by
Golden tests on DuckDB
Platforms
Databricks first, then Snowflake and Microsoft Fabric

Overview

What Late-arriving does.

Reprocess the right time windows when facts arrive after the period they belong to.

What it does

Late-arriving finds facts whose event date falls in a window that was already processed and reprocesses only the affected windows downstream. Past totals are corrected and nothing else moves.

It also handles the mirror case, a fact that arrives before its dimension row. The fact is linked to a placeholder member straight away and corrected to the real dimension version when that row arrives. In the meantime it is not dropped by an inner join.

Use it when

  • Mobile and offline events that sync hours or days after they happened.
  • Partner files that arrive days late.
  • Corrections and restatements from a source.
  • Any pipeline where event time and arrival time differ.

Use another shape when

Slots

Inputs, outputs and slots.

What the Late-arriving template reads, what it writes and the typed slots it asks you for. The agent fills most slots from your catalog and your request.

In

Inputs

  • New facts, each with an event time and an arrival time
  • The fact tables and aggregates already built
  • The dimension table, with its versions

Late-arriving

  1. Source
  2. Bronze
  3. Silver (used)
  4. Gold (used)
  5. Apps

Out

Outputs

  • Facts landed with the right dimension version, or the placeholder
  • Recomputed windows in the downstream tables
  • A report: windows touched, rows changed and facts older than the reprocess window

Template slots

6 slots · names and types may change before release

event_time_column
column
When the fact happened.
arrival_time_column
column
When the fact reached the pipeline.
reprocess_window
duration
How far back to reprocess. Facts older than this are reported, not silently dropped.
window_grain
enum
The size of the windows to recompute: day, week or month.
downstream_table
table
The aggregate or gold table to recompute for the affected windows.
placeholder_key
expression
The key of the unknown member that early facts are linked to.

Slot types are identifier, table, column, columns, expression, expressions, string, integer, boolean, duration and enum. What is a template slot.

Testing

Tested before it ships.

Every template comes with golden tests that run on DuckDB on your machine before anything reaches your platform.

How a run goes

The golden test delivers facts at the worst moments, then checks both what was corrected and what stayed still.

  1. 01Fixtures go in: small tables with the hard cases for the shape.
  2. 02The template runs on DuckDB on your machine, with no cluster and no warehouse compute.
  3. 03The result is compared with the expected output, row for row.
  4. 04Green can deploy. Red sends the agent back to the template.

What the Late-arriving test checks

  1. Case 01

    Deliver a fact two days late. The affected daily totals are corrected, and nothing outside the window changes.

  2. Case 02

    Deliver a fact whose dimension row has not arrived. It links to the placeholder and is not dropped.

  3. Case 03

    Deliver the dimension row afterward. The fact is relinked to the right version.

  4. Case 04

    Deliver a fact older than the reprocess window. It lands and is reported, with the reason totals did not change.

  5. Case 05

    Run the same delivery twice. Nothing changes the second time.

Each template is also judged per platform: yellow when it passes on DuckDB, green when it also passes on the real platform. Nothing is released, so no template has a green mark yet.

Platforms

Written for the platform you run.

The same shape has a different correct form on each platform. This is what the Late-arriving template uses on each one, and the platform facts that decide it.

Databricks

In development

The Databricks template runs MERGE into Delta fact tables, then uses replaceWhere on the affected partitions of the aggregates.

  • MERGE lands the new facts by key without duplicates. The batch's event dates then tell the template which partitions of the aggregate tables to recompute with replaceWhere.
  • Streaming pipelines need a lateness allowance. A streaming watermark drops events later than its allowance in stateful operations such as windowed aggregates, and a batch correction like this one is the usual way to catch what it dropped.
  • A placeholder member for early-arriving facts is an ordinary row in the dimension table with a reserved key.
  • Facts join to the dimension version that was valid at their event time, which is why this shape pairs with SCD Type 2.
Emolysis for Databricks

Snowflake

Planned

The Snowflake template uses a stream on the fact table to drive a task that recomputes the affected dates.

  • The stream reports which fact rows are new. The task derives the distinct event dates from them and recomputes only those dates.
  • A dynamic table recomputes affected results by itself, which handles late rows for simple aggregates. The template uses an explicit recompute when the reprocess window must be bounded and logged.
  • Deleting and inserting the affected dates in one transaction means dashboards never see a half-corrected day.
  • Unquoted identifiers fold to uppercase, so the event time and arrival time columns are matched the same way in every statement.
Emolysis for Snowflake

Microsoft Fabric

Planned

The Fabric template uses a notebook that finds the affected dates and rewrites those partitions.

  • The notebook compares the event dates in the new batch with the dates already processed, then rewrites those partitions of the lakehouse Delta tables.
  • A Data Pipeline runs it after each landing, and the reprocess window is a pipeline parameter.
  • Reports built on the corrected tables show the change once their semantic model next refreshes.
  • A warehouse target deletes and inserts the affected dates in one T-SQL transaction.
Emolysis for Microsoft Fabric

Whatever the platform, the template is tested on DuckDB first. The platform notes describe each product in general terms, so check the details against the version you run.

Mistakes

Mistakes it prevents.

The failures that most often hit a pipeline of this shape in production. Each one is a case in the template's golden test or a rule in its design.

  1. Late facts dropped by a watermark

    A streaming watermark discards events later than its allowance, and nobody sees them go. The template lands every fact and reports the ones outside the reprocess window.

  2. Aggregates that are never corrected

    The fact table is right and last week's dashboard is still wrong. The template recomputes the downstream windows the new facts touched.

  3. A fact joined to the wrong version of a dimension

    Joining on the current version rewrites history. The template joins on the version valid at the event time.

  4. Facts dropped by an inner join because their dimension row has not arrived yet

    The order exists, the customer record does not, and the join silently removes the order. The template links the fact to a placeholder and fixes the link later.

SQL sketch

The idea in SQL.

Land late facts, then recompute only the touched dates

Illustrative sketch in generic SQL. A reserved key of -1 stands for the unknown member. Each platform's template uses that platform's merge and partition-replace mechanisms.

Illustrative sketch · SQL
BEGIN;

-- 1. Land the new facts once. A fact whose customer is unknown yet links to the placeholder (-1).
INSERT INTO silver.orders (order_id, customer_key, amount, event_ts, arrived_at)
SELECT b.order_id, COALESCE(c.customer_key, -1), b.amount, b.event_ts, b.arrived_at
FROM staged.orders_batch AS b
LEFT JOIN gold.dim_customer AS c
  ON c.customer_id = b.customer_id
 AND b.event_ts >= c.valid_from AND b.event_ts < c.valid_to
WHERE NOT EXISTS (SELECT 1 FROM silver.orders AS o WHERE o.order_id = b.order_id);

-- 2. Recompute only the event dates this batch touched, inside the 7-day reprocess window.
--    Facts older than the window are reported separately, not dropped.
DELETE FROM gold.daily_revenue
WHERE revenue_date IN (SELECT DISTINCT CAST(event_ts AS DATE) FROM staged.orders_batch)
  AND revenue_date >= CURRENT_DATE - INTERVAL '7' DAY;

INSERT INTO gold.daily_revenue (revenue_date, revenue)
SELECT CAST(o.event_ts AS DATE), SUM(o.amount)
FROM silver.orders AS o
WHERE CAST(o.event_ts AS DATE) IN (SELECT DISTINCT CAST(event_ts AS DATE) FROM staged.orders_batch)
  AND CAST(o.event_ts AS DATE) >= CURRENT_DATE - INTERVAL '7' DAY
GROUP BY CAST(o.event_ts AS DATE);

COMMIT;

FAQ

Questions about Late-arriving.

What are late-arriving facts?

Late-arriving facts reach the pipeline after the period they belong to was already processed, such as an offline mobile event or a partner file delivered days late. If the pipeline ignores them, totals for past periods stay wrong until someone notices.

What is an early-arriving fact?

An early-arriving fact, sometimes called a late-arriving dimension, reaches the pipeline before the dimension row it refers to. An inner join would drop it. The usual fix is to link the fact to a placeholder member of the dimension and correct the link when the real row arrives, which is what the template does.

How far back should I reprocess?

As far back as facts can realistically arrive. Look at the gap between event time and arrival time in your data and pick a window that covers nearly all of it. The reprocess_window slot sets it, and facts beyond it are landed and reported, so you can see how often it is exceeded.

How is Late-arriving different from Backfill?

Backfill redoes a date range you choose. Late-arriving works out which windows the newly arrived facts affected and recomputes only those. Both replace partitions rather than append, so both are safe to repeat.

In development · Not released

Test first, then deploy.

The Late-arriving template is written for Databricks, Snowflake and Microsoft Fabric, and its golden tests pass on DuckDB. Nothing is released yet. Early access is by invitation, for teams that want to help shape it.