Glossary
Data engineering terms, in plain language.
From slowly changing dimensions to watermarks and golden tests: what each term means, and where Emolysis uses the idea.
New to the product? Start with the docs or the ten pipeline shapes.
A to Z
A
1 term
B
3 terms
Backfill
A backfill re-runs a pipeline for a past period, to load history, apply a fix to old data or reprocess data a source delivered again.
Also: back-fill · historical load · reprocessing
Bring your own key (BYOK)
Bring your own key means you supply the credentials for a service, here an AI model provider's key, and pay that provider directly.
Also: BYOK · own key · own API key
Bronze layer
The bronze layer holds raw data as it arrived from the source, with little or no change, plus load metadata such as the source file.
Also: bronze table · raw layer · landing layer
C
3 terms
Change data capture (CDC)
Change data capture records the inserts, updates and deletes made to a source table so downstream systems can apply only what changed.
Also: CDC · change feed · change data feed
Clustering
Clustering stores rows with similar values close together in a table's files, so queries that filter on those columns can skip most of the data.
Also: liquid clustering · clustering key
Coding agent
A coding agent is an AI system that writes, runs and fixes code toward a goal, using tools such as a file system, a terminal and tests.
Also: AI coding agent · code agent
D
8 terms
Data contract
A data contract is an agreement between the producer and consumers of a dataset about its columns, types, keys and meaning, checked automatically.
Also: dataset contract · schema contract
Data quality test
A data quality test checks that data meets an expectation: a key is unique and not null, a value is in range, a table is fresh.
Also: data test · data quality check · expectation
Databricks Asset Bundle
A Databricks Asset Bundle describes a project's jobs, pipelines and settings as files, so they deploy the same way to every environment.
Also: Asset Bundle · DAB · databricks.yml
Deduplication
Deduplication removes repeated records so each entity or event appears once, using a deterministic rule to choose which version to keep.
Also: dedupe · dedup · duplicate removal
Delta Lake
Delta Lake is an open table format that adds a transaction log to Parquet files, giving tables transactions, merges and time travel.
Also: Delta table · Delta format
Dimension
A dimension describes the things that facts are about, such as customers, products or dates, and holds the attributes reports group and filter by.
Also: dimension table · dim table
DuckDB
DuckDB is an open-source analytical database that runs inside another program or on a laptop, with no server to set up or manage.
Also: DuckDB engine · in-process analytics database
DuckLake
DuckLake is an open lakehouse format that keeps table metadata in a SQL database and the data as Parquet files on disk or object storage.
Also: DuckLake format · DuckDB lakehouse format
E
1 term
F
1 term
G
3 terms
Gold layer
The gold layer holds business-ready tables: metrics, aggregates and dimensional models at a declared grain, built for dashboards and reports.
Also: gold table · metrics layer · serving layer
Golden test
A golden test runs code on a fixed input and compares its output with a stored expected output; any difference fails the test.
Also: golden master test · golden file test · expected-output test
Grain
The grain of a table is what one row represents, such as one order line, one customer per day or one region per month, stated up front.
Also: granularity · level of detail
I
2 terms
Idempotency
A pipeline step is idempotent when running it twice with the same input leaves the same result as running it once, so retries are safe.
Also: idempotent · idempotent pipeline · safe re-run
Incremental load
An incremental load processes only data that is new or changed since the last run, instead of reloading the whole source every time.
Also: incremental processing · delta load · incremental refresh
L
3 terms
Lakehouse
A lakehouse stores data as open files in object storage and adds warehouse features: schemas, transactions, updates, deletes and governance.
Also: data lakehouse · lake house
Late-arriving records
Late-arriving records are facts that reach the pipeline after the period they belong to was already processed, so past totals need fixing.
Also: late-arriving facts · late data · late-arriving dimensions
Lineage
Lineage describes where data comes from and where it goes: which sources, tables and columns feed each table and report.
Also: data lineage · column lineage · provenance
M
3 terms
Medallion architecture
The medallion architecture organizes a lakehouse into bronze, silver and gold layers of increasing quality, each built from the one before.
Also: bronze silver gold · multi-hop architecture
Merge (upsert)
A merge, or upsert, writes a batch of rows into a table in one statement: existing keys are updated and new keys are inserted.
Also: upsert · MERGE INTO · merge statement
Model Context Protocol (MCP)
The Model Context Protocol is an open standard that lets editors and coding agents connect to external tools through a common interface.
Also: MCP · MCP server · MCP tool
N
1 term
O
2 terms
OneLake
OneLake is the single storage layer of Microsoft Fabric, where every workspace keeps its lakehouse and warehouse data, mostly as Delta tables.
Also: Fabric OneLake · Microsoft OneLake
Orchestration
Orchestration runs pipeline steps in the right order, on a schedule or when data arrives, with retries, dependencies and alerts.
Also: workflow orchestration · job scheduling · scheduler
P
3 terms
Partitioning
Partitioning splits a table into parts by the value of a column, most often a date, so queries and writes touch only the parts they need.
Also: table partitioning · partition pruning
Platform profile
In Emolysis, a platform profile is a structured description of a data platform that the agent loads when you connect a workspace.
Also: profile · platform description
Pull request check
A pull request check is an automated test that runs when code is proposed for merging and reports pass or fail on the pull request.
Also: PR check · status check · CI check
S
8 terms
SCD Type 1
A slowly changing dimension of Type 1 keeps only the latest value of each attribute and overwrites the old value in place.
Also: slowly changing dimension type 1 · SCD1 · overwrite dimension
SCD Type 2
A slowly changing dimension of Type 2 keeps the full history of a record, with a validity range for every version of it.
Also: slowly changing dimension type 2 · SCD2 · type 2 dimension
Schema evolution
Schema evolution is how a table's structure changes over time: new columns, widened types, and renamed or removed fields.
Also: schema drift · schema change
Semantic model
A semantic model describes data in business terms: the measures, the dimensions they can be sliced by, how tables join and the rules behind each number.
Also: semantic layer · metrics layer · business model
Silver layer
The silver layer holds cleaned, conformed data: one row per entity or event, consistent types and names, and duplicates removed.
Also: silver table · cleaned layer · conformed layer
Snapshot
A snapshot is a full copy of a source as it was at one point in time, either replacing the last copy or kept with its date.
Also: full refresh · periodic snapshot · snapshot table
SQL model
A SQL model is a single transformation kept as a file, usually one SELECT statement that defines one table or view in a project.
Also: model · SQL transformation
Surrogate key
A surrogate key is an identifier created by the data platform instead of taken from the source, usually a sequence number or a hash.
Also: synthetic key · generated key · warehouse key
T
1 term
U
1 term
W
1 term