Templates in development

Written for Databricks, Snowflake and Fabric and tested on DuckDB. None has run on a real platform yet, and none is released.

Dedupe / conform

Dedupe / conform: one row per key.

Turn raw bronze rows into clean silver rows: one row per key, chosen by a deterministic rule, with consistent types, names and time zones. Rows that fail a check go to quarantine, not into the void.

Template in development
Model and serve
Databricks first

Shape card

08 / 10

  1. Source
  2. Bronze (used)
  3. Silver (used)
  4. Gold
  5. Apps
Group
Model and serve
Works across
Bronze to Silver
Reads
Bronze rows
Writes
A silver table and a quarantine table
Slots
8 typed slots
Tested by
Golden tests on DuckDB
Platforms
Databricks first, then Snowflake and Microsoft Fabric

Overview

What Dedupe / conform does.

Turn raw bronze rows into clean silver rows: one row per key, with consistent types and names.

What it does

Dedupe / conform removes duplicate records with a deterministic rule, such as the latest by update time with a fixed tie-breaker. It casts columns to agreed types and normalizes names, time zones and codes.

Rows that fail a check, such as a null key or a value that cannot be cast, are set aside in a quarantine table with the reason. They are not dropped silently and not turned into nulls.

Use it when

  • Every source on its way from bronze to silver.
  • Several sources describe the same entity and must agree on its key and types.
  • After a source's schema drifts, for example a number that starts arriving as text.

Use another shape when

  • The raw rows must stay exactly as they arrived:

    01Ingest (batch)
  • The source already delivers a change feed with inserts, updates and deletes:

    03CDC merge
  • Every past version of a record must be kept, not just the latest:

    04SCD Type 2

Slots

Inputs, outputs and slots.

What the Dedupe / conform template reads, what it writes and the typed slots it asks you for. The agent fills most slots from your catalog and your request.

In

Inputs

  • One or more bronze tables, with load metadata
  • The key columns, the ordering for "latest wins" and the type map
  • The naming convention and the checks a row must pass

Dedupe / conform

  1. Source
  2. Bronze (used)
  3. Silver (used)
  4. Gold
  5. Apps

Out

Outputs

  • A silver table with exactly one row per key and consistent types
  • A quarantine table with the failed rows and the reason for each
  • A run record: rows in, rows out and rows quarantined

Template slots

8 slots · names and types may change before release

source_table
table
The bronze table to clean.
target_table
table
The silver table to write.
key_columns
columns
The columns that identify one entity.
order_columns
columns
The columns that decide which version wins. Together they must give a total order, so end with a tie-breaker.
type_map
expressions
The agreed type of each column.
naming_convention
enum
How column names are normalized, for example snake_case.
checks
expressions
Conditions a row must meet to land in silver, such as a key that is not null.
quarantine_table
table
Where rows that fail a check are set aside.

Slot types are identifier, table, column, columns, expression, expressions, string, integer, boolean, duration and enum. What is a template slot.

Testing

Tested before it ships.

Every template comes with golden tests that run on DuckDB on your machine before anything reaches your platform.

How a run goes

The golden test feeds the messy input this shape exists for and checks that the answer is the same on every run.

  1. 01Fixtures go in: small tables with the hard cases for the shape.
  2. 02The template runs on DuckDB on your machine, with no cluster and no warehouse compute.
  3. 03The result is compared with the expected output, row for row.
  4. 04Green can deploy. Red sends the agent back to the template.

What the Dedupe / conform test checks

  1. Case 01

    Duplicates with identical update times. The tie-breaker picks the same row on every run.

  2. Case 02

    A numeric column arrives as text in one batch. Rows are cast or quarantined as specified, and none becomes a silent null.

  3. Case 03

    Timestamps arrive in mixed time zones. They all land in one zone.

  4. Case 04

    Two column names differ only by case. They map to one column.

  5. Case 05

    A row with a null key goes to quarantine with the reason.

Each template is also judged per platform: yellow when it passes on DuckDB, green when it also passes on the real platform. Nothing is released, so no template has a green mark yet.

Platforms

Written for the platform you run.

The same shape has a different correct form on each platform. This is what the Dedupe / conform template uses on each one, and the platform facts that decide it.

Databricks

In development

The Databricks template uses window functions with a total order, QUALIFY in Databricks SQL and Delta constraints for checks.

  • ROW_NUMBER with QUALIFY keeps one row per key in a single statement. Add every ordering column and a final tie-breaker so the result is the same on every run.
  • Delta CHECK and NOT NULL constraints fail a write that violates them. The template uses them where a hard stop is right and a quarantine table where it is not.
  • Whether a failed cast raises an error or returns null depends on the compute and its ANSI setting, so the template uses try_cast explicitly and checks for the nulls it produces.
  • Silver tables live in Unity Catalog, so the schema's grants decide who may read the cleaned data and who may read the quarantine table.
Emolysis for Databricks

Snowflake

Planned

The Snowflake template uses QUALIFY with ROW_NUMBER.

  • Unquoted identifiers fold to uppercase, so naming rules have to account for case. Two bronze columns that differ only by case need quoted identifiers to stay apart.
  • TRY_CAST and the TRY_ conversion functions return null instead of raising an error. The template pairs them with a check that routes those rows to quarantine.
  • Timestamps come in three types, without a zone, with the session's zone and with an explicit offset. The template picks one zone-aware type for conformed columns.
  • A dynamic table can maintain the deduplicated result continuously, but the quarantine write needs a task or a stream.
Emolysis for Snowflake

Microsoft Fabric

Planned

The Fabric template uses window functions in Spark SQL on lakehouse tables.

  • Filtering on ROW_NUMBER in a subquery works in Spark SQL and in T-SQL warehouses alike. QUALIFY is not available everywhere, so the lakehouse template does not depend on it.
  • try_cast in Spark SQL and TRY_CAST in T-SQL make failed casts visible as nulls, which the checks then catch.
  • Delta tables in OneLake give the silver table the same transactional behavior as on Databricks.
  • A Data Pipeline runs the notebook after the bronze load, and the quarantine table is an ordinary lakehouse table that a report can read.
Emolysis for Microsoft Fabric

Whatever the platform, the template is tested on DuckDB first. The platform notes describe each product in general terms, so check the details against the version you run.

Mistakes

Mistakes it prevents.

The failures that most often hit a pipeline of this shape in production. Each one is a case in the template's golden test or a rule in its design.

  1. Dedupe results that change between runs because the ordering is incomplete

    Two rows with the same update time are ordered by chance, and the survivor changes from run to run. The template requires a total order with a fixed tie-breaker.

  2. Silent nulls from failed casts

    A value that cannot be cast becomes null and flows into reports unnoticed. The template checks for it and quarantines the row, or fails, as the slots say.

  3. Identifiers that differ only by case

    Order_ID and order_id are two columns in one engine and one column in another. The naming convention maps them deliberately and the test includes a collision.

  4. Timestamps in mixed time zones

    Events from two sources are compared as if they shared a zone and appear hours apart. The template converts every timestamp to one zone on the way into silver.

SQL sketch

The idea in SQL.

One deterministic row per key, with a quarantine

Illustrative sketch in generic SQL. Databricks and Snowflake templates can use QUALIFY instead of the subquery. Names are placeholders.

Illustrative sketch · SQL
-- 1. Rows that fail a check are set aside with the reason, not dropped.
INSERT INTO silver.customers_quarantine (customer_id_raw, signup_date_raw, reason, _source_file)
SELECT b.customer_id, b.signup_date,
       CASE WHEN b.customer_id IS NULL THEN 'null key' ELSE 'signup_date cannot be cast' END,
       b._source_file
FROM bronze.customers AS b
WHERE b.customer_id IS NULL
   OR (b.signup_date IS NOT NULL AND TRY_CAST(b.signup_date AS DATE) IS NULL);

-- 2. One row per key: latest update wins; a total order makes the result repeatable.
INSERT INTO silver.customers (customer_id, country_code, signup_date, updated_at)
SELECT customer_id, country_code, signup_date, updated_at
FROM (
  SELECT
    CAST(b.customer_id AS BIGINT)         AS customer_id,
    UPPER(TRIM(b.country_code))           AS country_code,
    TRY_CAST(b.signup_date AS DATE)       AS signup_date,
    TRY_CAST(b.updated_at AS TIMESTAMP)   AS updated_at,
    ROW_NUMBER() OVER (
      PARTITION BY b.customer_id
      ORDER BY TRY_CAST(b.updated_at AS TIMESTAMP) DESC, b._source_file DESC, b._row_number DESC
    ) AS rn
  FROM bronze.customers AS b
  WHERE b.customer_id IS NOT NULL
    AND (b.signup_date IS NULL OR TRY_CAST(b.signup_date AS DATE) IS NOT NULL)
) AS ranked
WHERE rn = 1;

FAQ

Questions about Dedupe / conform.

How do I deduplicate rows in SQL deterministically?

Number the rows within each key with ROW_NUMBER, ordered by the columns that decide which version wins, such as the update time, and then by a fixed tie-breaker such as the source file and row number. Keep the row numbered 1. Without the tie-breaker, rows with equal update times are ordered by chance and the result can change between runs.

What is the difference between deduping and conforming?

Deduping decides which of several records for the same entity to keep. Conforming makes the surviving records consistent: the same types, names, codes and time zones across sources. They belong in one shape because the order matters, and both turn bronze into silver.

What belongs in quarantine, and what should fail the run?

Quarantine suits individual bad rows, such as a null key or a value that cannot be cast, when the rest of the batch is fine. A hard failure suits problems with the whole batch, such as a missing column. The template takes the choice per check as a slot, so it is explicit.

What does bronze to silver mean?

In a medallion architecture, bronze holds raw data as it arrived and silver holds cleaned, conformed data. Moving from one to the other is where duplicates are removed, types are fixed and bad rows are set aside. That step is the Dedupe / conform shape.

In development · Not released

Test first, then deploy.

The Dedupe / conform template is written for Databricks, Snowflake and Microsoft Fabric, and its golden tests pass on DuckDB. Nothing is released yet. Early access is by invitation, for teams that want to help shape it.