Pipeline 01
A source database to a dashboard
Land the extract in bronze, clean it into silver, then build the gold table that a dashboard reads.
Templates in development
Written for Databricks, Snowflake and Fabric and tested on DuckDB. None has run on a real platform yet, and none is released.
Pipeline shapes
Ten pipeline shapes.
Most production pipelines follow one of about ten shapes, and each shape has a known correct form on each platform. Emolysis starts from a tested template for the shape and the platform you use, instead of writing a pipeline from scratch. Each template comes with test fixtures and expected outputs that run on DuckDB on your machine before anything reaches your platform.
Templates for all ten shapes are written for Databricks, Snowflake and Microsoft Fabric, and their golden tests pass on DuckDB. None has run on a real platform account yet, and none is released.
The lineup
The ten pipeline shapes
01
Load
Ingest (batch)
Land files, extracts and database tables in bronze tables, typed and checked on load.
- Source (used)
- Bronze (used)
- Silver
- Gold
- Apps
Files, extracts, source tablestoTyped bronze tables
Template in development
02
Load
Incremental append
Append only the rows that arrived since the last run, tracked by a watermark.
- Source (used)
- Bronze (used)
- Silver
- Gold
- Apps
An append-only sourcetoNew rows in the target
Template in development
03
Sync and history
CDC merge
Apply inserts, updates and deletes from a change feed to a target table.
- Source (used)
- Bronze (used)
- Silver (used)
- Gold
- Apps
A change feedtoA table in its current state
Template in development
04
Sync and history
SCD Type 2
Keep the full history of a dimension, with a validity range for every version.
- Source
- Bronze
- Silver (used)
- Gold
- Apps
Current dimension recordstoA dimension with every version
Template in development
05
Load
Snapshot
Copy a source in full on a schedule and keep dated copies for a set retention period.
- Source (used)
- Bronze (used)
- Silver
- Gold
- Apps
The whole source, each runtoA replaced table or dated copies
Template in development
06
Repair
Backfill
Re-run a pipeline for a past date range, one partition at a time, without touching the rest.
- Source
- Bronze (used)
- Silver (used)
- Gold (used)
- Apps
A pipeline and a date rangetoReplaced partitions
Template in development
07
Repair
Late-arriving
Reprocess the right time windows when facts arrive after the period they belong to.
- Source
- Bronze
- Silver (used)
- Gold (used)
- Apps
Facts after their periodtoCorrected windows
Template in development
08
Model and serve
Dedupe / conform
Turn raw bronze rows into clean silver rows: one row per key, with consistent types and names.
- Source
- Bronze (used)
- Silver (used)
- Gold
- Apps
Bronze rowstoA silver table and a quarantine table
Template in development
09
Model and serve
Aggregation / gold
Build the metrics tables that dashboards and reports read, at a declared grain.
- Source
- Bronze
- Silver (used)
- Gold (used)
- Apps
Conformed silver tablestoGold metrics tables
Template in development
10
Model and serve
Reverse ETL
Push curated gold data back into business applications and operational databases, safe to re-run.
- Source
- Bronze
- Silver
- Gold (used)
- Apps (used)
Gold tablestoBusiness applications and databases
Template in development
Not sure which
Start from the source.
Describe what the source does, such as files arriving, rows changing or facts showing up late, and see which shape fits.
Which shape fitsBeyond one shape
Shapes in a row.
Most real pipelines chain several shapes, from the first landing to the dashboard. See four common chains.
How shapes combineWhich shape fits
Which shape fits?
Start from what your source does, not from the name of a pattern. Each row links to the shape that handles it.
- Files, extracts or tables arrive in batches and must be landed once.01Ingest (batch)
- The source only ever adds rows, and each run should read just the new ones.02Incremental append
- Rows in the source are updated and deleted, and the target must follow.03CDC merge
- Reports must show the value that was true at the time, not today's value.04SCD Type 2
- The source has no change tracking, and you want a full copy on a schedule.05Snapshot
- A past period must be recomputed without touching the rest of the table.06Backfill
- Facts arrive after the period they belong to, and past totals are now wrong.07Late-arriving
- Raw rows hold duplicates, mixed types and inconsistent names or time zones.08Dedupe / conform
- A dashboard or report needs one trusted table with declared metrics.09Aggregation / gold
- Computed values must reach a sales, support or marketing tool.10Reverse ETL
Shapes in a row
Real pipelines are shapes in a row.
Landing, cleaning, keeping history and serving are different jobs, so a production pipeline is usually several shapes in a row. Emolysis composes them.
Pipeline 02
A table that changes, with its history
Mirror the source's current state, keep every version of the dimension, then report on facts joined to the version that was true at the time.
Pipeline 03
Events that arrive late
Append new events with a watermark, correct the windows that late events touched, then refresh the metrics.
Pipeline 04
Fix the past, then send it on
Replay a date range after a fix, rebuild the gold table for it, then push the corrected values into the tools that use them.
Templates
Every template is tested before it deploys.
A template is a known-correct form of one shape on one platform, with the tests that prove it. The agent fills it in instead of writing a pipeline from scratch.
The path
From a request to a deploy
- 01
Pick the shape
The agent matches your request to one of the ten shapes and offers only the templates for your connected platform.
- 02
Fill the slots
It fills the slots from your catalog and your request, and asks you only for what is missing.
- 03
Prove it on DuckDB
The golden tests run on your machine. A red result sends the agent back to the template.
- 04
Deploy in native units
The agent deploys only after a green run, as your platform's own unit, such as a Databricks Asset Bundle.
Inside a template
Five parts, one folder
Typed slots
Named placeholders such as the business key or the watermark column. Each slot has a type, such as column, columns, table, duration or enum, so a value cannot land in the wrong place.
Test fixtures
Small input tables that include the hard cases for the shape: duplicates, late rows, a malformed file, several changes to one key.
Expected outputs
The exact tables a correct run produces. A golden test compares the actual output with them, row for row.
A cost note
A short note on what a run of the template costs on the platform, so the price of a pipeline is no surprise.
Platform constraints
The platform features and versions the template needs, so it is offered only where it can run.
Each template is judged per platform: yellow when it passes on DuckDB, green when it also passes on the real platform. Nothing is released, so no template has a green mark yet.
Platforms
Written for the platform you run.
Each shape has its own correct form on Databricks, Snowflake and Microsoft Fabric. Databricks comes first. Snowflake and Fabric are planned.
Databricks
In development
The first target. Writes for Delta tables and Unity Catalog and deploys as Databricks Asset Bundles.
- Tables
- Delta tables governed by Unity Catalog
- Merge
- MERGE INTO, with Delta Change Data Feed where it is on
- Incremental
- Auto Loader and Lakeflow Declarative Pipelines
- Deploys as
- Databricks Asset Bundles
Snowflake
Planned
Writes for streams, tasks and dynamic tables, with Snowflake's own identifier rules.
- Tables
- Native Snowflake tables or Iceberg tables
- Merge
- MERGE fed by streams, and dynamic tables
- Incremental
- Streams and tasks
- Deploys as
- SQL that creates tables, streams, tasks and dynamic tables
Microsoft Fabric
Planned
Writes for lakehouses in OneLake and deploys notebooks to your Fabric workspace.
- Tables
- Delta tables in OneLake lakehouses
- Merge
- MERGE on lakehouse Delta tables
- Incremental
- Notebooks on a pipeline schedule, and Dataflow Gen2
- Deploys as
- A PySpark notebook in your workspace
PostgreSQL is in development as a source, and SQL Server is planned as one; landing their data through Ingest (batch), Incremental append and CDC merge is planned. Every template is tested first on DuckDB, your local engine.
FAQ
Questions about shapes.
What are pipeline shapes?
A pipeline shape is a recurring kind of data pipeline with a known correct form, such as loading files into bronze, merging a change feed, or keeping history for a dimension. Data engineering is more repetitive than general software: about ten shapes cover most production work. For each shape and platform, Emolysis has a template with named slots, test fixtures and expected outputs.
Read the full answerWhat if my pipeline does not fit one of the ten shapes?
Most real pipelines are several shapes in a row: ingest to bronze, dedupe and conform to silver, SCD Type 2 for a dimension, then aggregation to gold. Emolysis composes shapes like that. When a request truly does not fit any shape, the agent says so rather than improvising a pipeline it cannot test. You can still write that part yourself in the same project, with your own tests.
Read the full answerWhat is a golden test?
A golden test runs a pipeline on a fixed input and compares its output with an expected output, row for row. If they differ, the test fails. Each Emolysis template ships with golden tests built for its shape. The tests run on DuckDB on your machine. A red result sends the agent back to the template, and the agent deploys only after a green run.
Read the full answerWhy test on DuckDB instead of on my platform?
Speed and cost. DuckDB runs on your machine, starts instantly and needs no cluster, so a golden test takes seconds and spends no warehouse compute. That makes it practical to run the tests on every save, not only before a release. A template is only called ready on a platform after its cases also pass on that platform, and none has yet.
Read the full answerIn development · Not released
Start from a template, not a blank page.
Templates for the ten shapes are written for Databricks, Snowflake and Microsoft Fabric, and their golden tests pass on DuckDB; none has run on a real platform yet. Early access is by invitation, for teams that want to help shape them.