Status
In private development. Early access by invitation.
A pipeline shape is a recurring kind of pipeline with a known correct form. Emolysis is built around ten of them, because together they cover most production data engineering. When you ask for a pipeline, the agent starts by choosing the shape, not by writing code.
The ten shapes
| Shape | What it covers | Golden test fixture |
|---|---|---|
| Ingest (batch) | Files, APIs or databases into bronze, typed, deduplicated on load | Three files, one malformed |
| Incremental append | Watermark-based append of new rows | Two runs that overlap |
| CDC merge | Inserts, updates and deletes from a change feed | A mix of inserts, updates, deletes |
| SCD Type 2 | History with validity ranges | An attribute change across three runs |
| Snapshot | Full refresh with retention | Two snapshots |
| Backfill | Partitioned re-run of a date range | A gap in the middle of the range |
| Late-arriving | Reprocessing windows when facts arrive late | A fact two days late |
| Dedupe / conform | Keys, types and naming into silver | Duplicates and type drift |
| Aggregation / gold | Metrics tables for BI | Input with known totals |
| Reverse ETL | Gold data pushed to applications or databases | The same push sent twice |
Each shape has its own page under /shapes, with platform notes and the mistakes its template prevents.
How the agent chooses
The agent reads your request and your catalog, then picks the shape whose purpose matches. Some requests name it directly ("an SCD2 on customers"). Others imply it. The agent tells you which shape it picked and why, and you can change it.
| You might ask | The shape it points to |
|---|---|
| "Load the files that land in this folder every night" | Ingest (batch) |
| "Add the new events every hour" | Incremental append |
| "Keep this table in step with the source, deletes included" | CDC merge |
| "We need to know which region a customer was in when they ordered" | SCD Type 2 |
| "Keep a copy of the ledger as of each month end" | Snapshot |
| "Reload the first quarter after the source fix" | Backfill |
| "Some partner files arrive days late" | Late-arriving |
| "One clean row per customer, with the same types everywhere" | Dedupe / conform |
| "Monthly revenue by region" | Aggregation / gold |
| "Send the churn list to the CRM every morning" | Reverse ETL |
Shapes combine
Real pipelines are usually several shapes in a row. A typical chain lands a source with Ingest (batch), cleans it with Dedupe / conform, keeps a dimension's history with SCD Type 2, and builds metrics with Aggregation / gold. This follows the medallion architecture: bronze, silver and gold layers.
Emolysis builds each step from its own template, with its own tests, and connects them in one pipeline. A failure in one step points at one template and one test, not at the whole chain.
Slots
Each shape has a slot schema: the named, typed values its template needs, such as the business key for SCD Type 2 or the watermark column for Incremental append. The agent fills what it can from your catalog and your request, and asks you only for what is missing. See Templates and slots.
Choosing between neighbors
A few shapes are easy to confuse. These rules of thumb are how the agent separates them:
- Incremental append or CDC merge. If source rows are never updated after they are written, append. If they can change or be deleted, merge. See incremental load and change data capture.
- SCD Type 2 or an overwrite. If reports must show the value that was true at the time, keep history. If only the latest value matters, overwrite in place (an SCD Type 1 update, which the SCD Type 2 template also supports for columns you do not track).
- Snapshot or incremental. If the source has no reliable change tracking and is small enough to copy in full, snapshot it.
- Backfill or late-arriving. A backfill replaces a date range you name. Late-arriving reprocesses only the windows that new facts touched.
When nothing fits
If a request does not match any shape, the agent says so rather than improvise a pipeline it cannot test. You can write that part yourself in the project, with your own golden tests, or write a template of your own in the public template format.
Status
The library has a template for each shape on Databricks, Snowflake and Microsoft Fabric. Their golden tests pass on DuckDB; none has passed on a real platform yet.