Glossary

Databricks Asset Bundle

Also known as: Asset Bundle · DAB · databricks.yml

Definition

A Databricks Asset Bundle describes a project's jobs, pipelines and their settings as files, with a databricks.yml at the root, so they can be versioned, reviewed and deployed to different environments the same way every time. Bundles bring software practices such as pull requests and CI to Databricks work.

For example, a bundle for a daily orders pipeline might hold a databricks.yml that names the project and its targets, a resources/ folder with the job definition and its schedule, and the notebooks or Python files the job runs. The same files deploy to a development target, where each engineer gets their own copy of the job, and to a production target that runs under a service principal. Commands such as databricks bundle validate and databricks bundle deploy -t prod check the files and then create or update the job, so what runs in production is what was reviewed.

Targets keep the differences between environments in one place: the workspace, the catalog and schema names, the compute settings, and who the job runs as. Variables fill in values that change per target, so the job definition does not need copies. A common mistake is to change a job by hand in the workspace after a deploy. The next bundle deploy overwrites it, which is the point, but the hand change is lost if it was never committed.

Bundles also suit review. A pull request that changes a schedule, a cluster size or a table name shows the change as a diff in a YAML file, which reviewers can read and checks can test. Related terms: Unity Catalog, where the bundle's tables live, and orchestration, which runs the jobs a bundle defines. The Databricks platform page shows where bundles fit in Emolysis's Databricks support.

Emolysis deploys Databricks pipelines as Asset Bundles, the deployment unit in its Databricks platform profile. See Deploying and pull request checks. Because a bundle is a set of files, going back to an earlier version means deploying an earlier commit.