Glossary

Apache Iceberg

Also known as: Iceberg · Iceberg table

Definition

Apache Iceberg is an open table format for large analytic tables. It tracks a table's files and snapshots in metadata, which gives transactions, schema and partition evolution, and time travel across engines. Snowflake supports Iceberg tables alongside its native tables, and many engines can read and write it.

For example, an Iceberg table keeps a metadata file that points to the current snapshot, a manifest list for that snapshot, and manifests that list the data files with statistics such as the lowest and highest value of each column. A query engine reads that metadata, skips data files whose ranges cannot match the filter, and reads only the rest. A write creates new data files, new manifests and a new snapshot, then moves the table's pointer to the new metadata in one atomic step through the catalog.

Iceberg's partitioning is hidden: a table can be partitioned by the day of a timestamp column, and queries filter on the timestamp itself without knowing the partition scheme, which can be changed later without rewriting old data. Schema changes such as adding, renaming or reordering columns are tracked by column id, so a rename does not break old files.

An Iceberg table needs a catalog that knows where its current metadata is, such as a REST catalog, a Hive metastore or a platform's own catalog, and engines must use the same catalog to share a table. Old snapshots stay until they are expired, which bounds time travel and storage, much as with Delta Lake. See also lakehouse, partitioning and schema evolution.

Iceberg is developed as an Apache Software Foundation project, and engines such as Spark, Trino, Flink and DuckDB support it, which is why it is often chosen where several tools share tables. See also DuckDB and the Snowflake platform page.

Iceberg is one of the table formats in Emolysis's drafted Snowflake platform profile, and DuckDB can read Iceberg tables for local samples. Compare Delta Lake. Because several engines can share one Iceberg table, it is often chosen where data is read by more than one tool.