DuckLake 1.0: A New Data Lake Format with SQL Catalog Metadata

The data lake landscape is evolving with DuckDB Labs' release of DuckLake 1.0, a novel approach to data lake formatting that stores table metadata directly in a SQL database rather than distributing it across many files in object storage. This production-ready release represents a significant departure from established lake formats like Apache Iceberg, Delta Lake, and Apache Hudi, which primarily store metadata as files in object storage, often supplemented by catalog services.

The Problem with File-Based Metadata

Traditional data lake formats face several challenges with file-based metadata storage. According to the DuckDB team, this approach leads to complex coordination, slow metadata operations, and the proliferation of small files in object storage—a problem commonly known as "the small file problem." When dealing with numerous small files, object storage systems struggle with performance, and metadata operations become increasingly expensive as the number of files grows.

"File-based metadata in lake formats leads to complex coordination, slow metadata operations, and many small files in object storage," explains the DuckDB team. This observation forms the foundation of DuckLake's design philosophy.

DuckLake's SQL-Based Approach

DuckLake addresses these challenges by storing metadata directly in a SQL database. This approach was first articulated in the "DuckLake manifesto" published a year ago, which argued that lakehouse metadata should be centralized in a database rather than spread across many files.

"We are happy to announce DuckLake v1.0, almost a year after we released our first sketch of the specification," the team writes. "This is a production-ready release with guaranteed backward-compatibility. DuckLake v1.0 ships a stable specification, a feature-rich and fast reference implementation (the DuckDB ducklake extension), as well as a roadmap for future development."

The first implementation of DuckLake is available as a DuckDB extension, making it accessible to DuckDB users while maintaining compatibility with other data processing engines.

Key Features of DuckLake 1.0

DuckLake 1.0 introduces several features designed to improve lakehouse operations and performance:

Data Inlining

One of DuckLake's flagship features is data inlining, which enables performing small insert, delete, and update operations directly in the catalog database, avoiding the creation of new files.

"Data inlining is one of the flagship features of DuckLake," the team notes. "It basically enables performing small insert, delete and update operations in the catalog database, avoiding the proliferation of 'the small file problem'. DuckLake v1.0 brings full inlining of updates and deletes. This feature is now on by default with a default threshold of 10 rows."

This approach significantly reduces the overhead associated with small changes to datasets, which can be particularly problematic in traditional data lake formats where even minor updates might require rewriting entire files or creating numerous small files.

Sorted Tables

DuckLake introduces sorted tables to speed up filtered queries. By maintaining data in sorted order, the system can more efficiently locate and retrieve relevant data, reducing the computational cost of query operations, especially those involving range queries or filters.

Bucket Partitioning

For datasets with high-cardinality columns, DuckLake offers bucket partitioning. This technique distributes data across a fixed number of buckets based on the values in specified columns, improving query performance for operations that filter on these columns.

Geometry Data Types

DuckLake provides improved support for geometry data types, making it more suitable for geospatial applications and analyses. This enhancement expands the range of use cases for which DuckLake can be effectively employed.

Deletion Vectors

The format includes deletion vectors compatible with Iceberg, allowing for efficient handling of deleted records without requiring physical data removal. This feature ensures compatibility with existing tools and workflows that expect Iceberg-style deletion mechanisms.

Ecosystem Integration

DuckLake is designed to integrate with the broader data processing ecosystem. Clients are available for several popular data processing frameworks:

• Apache DataFusion
• Apache Spark
• Trino
• Pandas

For organizations preferring a managed solution, MotherDuck offers a hosted DuckLake service that manages the catalog database and storage infrastructure, reducing operational overhead.

Community Response

The release has generated interest in the data engineering community. On Hacker News, Alexander Dahl, a data platform engineer, commented: "Very exciting! The numbers seem to crush Iceberg. Has anyone tried it out for 'real' workloads?"

Some users have requested additional features, such as first-class support for the SMB protocol to better integrate with enterprise Windows environments. As one Reddit user noted: "A lot of enterprises still rely on SMB on-premises."

Future Roadmap

The DuckLake team has outlined a clear roadmap for future development:

• DuckLake v1.1 will introduce improvements such as variant inlining across catalogs and multi-deletion vector Puffin files.
• DuckLake v2.0 is planned to include Git-like branching for datasets and built-in role-based permissions, further enhancing its capabilities as a data lake format.

For those interested in exploring use cases and libraries, the awesome-ducklake repository provides curated resources.

DuckLake 1.0 is available on GitHub under an MIT license, encouraging community adoption and contribution.

Conclusion

DuckLake 1.0 represents a significant innovation in the data lake space by centralizing metadata in a SQL database rather than distributing it across files. This approach addresses several persistent challenges with traditional data lake formats, particularly the small file problem and slow metadata operations. With its production-ready status, comprehensive feature set, and clear roadmap, DuckLake is positioned to offer a compelling alternative for organizations seeking more efficient data lake solutions.