In private development
Emolysis is in private development. Connecting Databricks, the catalog mirror, sampling and deploying as Asset Bundles are built and tested on simulated Databricks services; none has run on a real workspace yet. There is nothing to install, and early access is by invitation.
Databricks
Emolysis for Databricks.
Databricks is the first platform Emolysis targets. It writes pipelines for Delta tables and Unity Catalog, proves them on your machine and deploys them as Databricks Asset Bundles.
In development
Connect
OAuth or a personal access token
Sample
Delta tables through a SQL warehouse
Prove
Golden tests on DuckDB
Deploy
A Databricks Asset Bundle
Overview
What it does on Databricks.
Databricks is the first platform Emolysis targets: connecting, sampling, testing and deploying, end to end.
When you connect a workspace, Emolysis loads a Databricks profile: the dialects, table format, merge idiom, incremental tools, deployment unit, compute and known traps of Databricks. The agent uses it on every request, and only Databricks templates are offered in that project.
Every request follows the same path. The agent picks one of the ten pipeline shapes, fills the template written for Databricks and runs its golden tests on DuckDB on your machine. The agent deploys only after a green run, and what deploys is a Databricks Asset Bundle that runs on your workspace's compute, under your account.
Databricks comes first because it offers APIs for every step of that loop: OAuth sign-in, Unity Catalog, sampling Delta tables, running jobs and deploying bundles. The profile format is the same for every platform, so this work carries over to Snowflake and Microsoft Fabric.
Platform profile
What the profile captures.
What Emolysis loads when you connect a Databricks workspace, so the agent writes code the way Databricks expects.
- Dialects
- Databricks SQL and PySpark
- Table format
- Delta tables governed by Unity Catalog
- Merge idiom
- MERGE INTO, with Delta Change Data Feed where it is on
- Incremental tools
- Auto Loader and Lakeflow Declarative Pipelines, including its change-data APIs
- Deployment unit
- Databricks Asset Bundles (databricks.yml)
- Compute
- Jobs, SQL warehouses and Databricks Connect
- Sign-in
- OAuth user-to-machine, or a personal access token
- Known traps
- Differences between serverless and classic compute, and Unity Catalog permissions
- Sampling
- Reading Delta tables through a SQL warehouse or Unity Catalog volumes
Connect
Connecting a workspace.
How connecting Databricks works, step by step.
Choose Databricks
In the app, choose Connect and then Databricks. From a terminal, the command is emo connect databricks; an editor or coding agent that supports MCP calls the platform_connect tool.
Sign in to Emolysis
If you are not signed in yet, the app opens your browser, where you sign in with your GitHub, Microsoft or Google account. From a terminal, you confirm a short code there.
Authorize with Databricks
You sign in with Databricks' own OAuth flow in the browser; a personal access token is for automation, such as CI and scripts. The credentials are kept in your machine's credential store and never sent to Emolysis.
Pick a workspace and compute
Emolysis lists the workspaces your account can reach. You pick one, then the SQL warehouse or cluster that samples and deploys should use.
Catalog mirror
Emolysis reads the catalogs, schemas, tables and columns your account can see into a local mirror, so the agent can find sources without a query per request. Metadata only; no table data is copied.
Sample on request
When you ask for a sample, Emolysis reads a small slice of a table through your SQL warehouse into DuckDB on your machine. That query runs on your platform; the tests that follow do not.
emo connect databricksDeploy
Deploys as an Asset Bundle.
A green run deploys as a Databricks Asset Bundle that you can review in a pull request, version and promote between environments. It runs on your workspace's compute, under your account.
Tested first
The agent runs lint and the golden tests before it deploys, and stops on a red test.
Tagged for Databricks
Every generated file carries its platform and profile version, so it is never deployed to another platform without an explicit translation step.
Translated at deploy time
Pipelines are written once in a canonical form and translated into Databricks SQL when you deploy, with the platform-specific parts of the template rendered for Databricks.
A bundle you own
The bundle's databricks.yml and its jobs or pipelines are files in your repository. Going back to an earlier version means deploying an earlier commit.
Reviewed in a pull request
emo check runs in your CI on the pull request: it lints the project, runs the golden tests and reports a cost estimate. Made a required check, a green run becomes the deploy gate.
Your compute, your account
The pipeline runs on your workspace's compute, under your account, on the schedule the bundle defines. Unity Catalog permissions still decide what it can read and write.
Known traps
Traps the profile knows.
Four mistakes that are easy to make on Databricks, and what the Databricks profile and its lint rules do about them.
Serverless and classic compute
Code that works on classic compute can behave differently on serverless compute. The profile records the difference, so lint rules flag code that depends on it before anything runs.
Unity Catalog permissions
Permissions decide what you, and the jobs you deploy, can read and write, and they are a common source of deploy failures. The Databricks lint rules flag them before anything runs.
One row per key in a merge
A MERGE INTO fails, or picks an arbitrary row, when a key appears twice in one batch. The CDC merge template applies only the latest change for each key, in order.
Time travel is not business history
Delta time travel and file retention settings decide how long old files stay. The Snapshot template keeps dated snapshots, so history does not depend on them.
Shapes
Ten shapes on Databricks.
Each shape has a template written for Databricks, with fixtures and expected outputs.
- Ingest (batch)Auto Loader or COPY INTO, writing Delta tables governed by Unity Catalog.
- Incremental appendAuto Loader or Structured Streaming with an available-now trigger, or a watermark filter on a Delta table, scheduled as a job.
- CDC mergeMERGE INTO on Delta tables, with Delta Change Data Feed as a source where it is on, or the change-data APIs in Lakeflow Declarative Pipelines.
- SCD Type 2MERGE INTO on Delta tables, or the SCD Type 2 option of the change-data APIs in Lakeflow Declarative Pipelines.
- SnapshotOverwrite or partition replace on Delta tables. Delta time travel and file retention settings decide how long old files stay.
- BackfillreplaceWhere or INSERT OVERWRITE on Delta partitions, run as a job with parameters.
- Late-arrivingMERGE into Delta fact tables, then replaceWhere on the affected partitions of aggregates. Streaming watermarks need a lateness allowance.
- Dedupe / conformWindow functions with a total order, QUALIFY in Databricks SQL, and Delta constraints for checks.
- Aggregation / goldDelta gold tables or materialized views, governed in Unity Catalog.
- Reverse ETLA scheduled job that reads the gold Delta table and can use Change Data Feed to find changed rows.
All ten shapes, with their templates and golden tests, are on the pipeline shapes page.
Read next
Questions
Databricks questions.
Answers about Databricks and how it fits. More in the FAQ.
Why is Databricks first?
Databricks offers a complete set of APIs for everything Emolysis needs to do end to end: sign in with OAuth, read Unity Catalog, sample Delta tables, run jobs and deploy Asset Bundles. That makes it the best place to prove the whole loop, from a question to a tested, deployed pipeline and a shared dashboard. The platform profile format is the same for every platform, so the work on Databricks carries over.
What permissions does Emolysis need on Databricks?
Emolysis acts with the permissions of the account you sign in with, no more. To work end to end, that account needs to read the catalogs and schemas you work with, use a SQL warehouse, create jobs, write files in its own workspace folder, and read and write the pipeline's tables in Unity Catalog. For a first try, use a development workspace or a dedicated catalog and schema, so nothing you test touches production.
Does testing spend Databricks compute?
Testing on a local sample spends no warehouse compute: golden tests run on DuckDB on your machine. Reading the sample from Databricks is a query on your SQL warehouse, so that step runs on your compute. Your workspace runs what you deploy.
Where are my Databricks credentials kept?
In your operating system's credential store, on your machine. You sign in with Databricks' own OAuth flow in the browser; a personal access token is for automation, such as CI and scripts. Credentials are never sent to Emolysis, and Emolysis's services never connect to your data platform.
Can I review what Emolysis deploys before it runs?
Yes. A deploy produces a Databricks Asset Bundle: files in your repository that you can read, review in a pull request and version like any other code. Deploying is a step you take; the agent does not deploy on its own.
Does Emolysis replace Unity Catalog?
No. Unity Catalog stays your catalog of record, and your tables, jobs and permissions stay on Databricks. Emolysis keeps a small local lakehouse in each project for samples and tests, as a development copy, not a replacement.
Nothing is released yet
Start with Databricks.
Early access is by invitation, starting with data engineering teams on Databricks.