Technologies tagged with "Lakehouse"

Apache ORC is a columnar file format built for the Hadoop and Hive ecosystem: per-column min/max statistics and bloom filters for skipping data, support for ACID transactions and snapshot isolation, and native handling of Hive's compound types.

Vortex is an extensible columnar file format written in Rust and incubating at the Linux Foundation, claiming order-of-magnitude gains over Parquet on random access and scans at comparable compression, with an encoding system designed to be extended rather than frozen into the specification.

Nimble is Meta's Apache-2.0 columnar file format, built as a replacement for Parquet and ORC on workloads with thousands of columns: FlatBuffers metadata, block rather than stream encoding, and an encoding layer decoupled from the file layout so new schemes can be added and composed.

Unity Catalog is an Apache-2.0 catalog hosted by the LF AI & Data Foundation that governs Delta, Iceberg, and Hudi tables alongside files, functions, and AI models, speaking both the Hive metastore API and the Iceberg REST catalog API.

Project Nessie is an Apache-2.0 transactional catalog for data lakes with git-like semantics: Iceberg tables and views are versioned on branches and tags, changes across multiple tables commit atomically, and engines connect through the Iceberg REST protocol.

Lakekeeper is an Apache-2.0 Apache Iceberg REST catalog written in Rust: a single binary with multi-tenant warehouses created at runtime, fine-grained authorization through OpenFGA or Cedar, vended credentials for S3, ADLS and GCS, and change events emitted to Kafka or NATS.

The Lakehouse runtime catalog — until April 2026, BigLake metastore — is Google Cloud's fully managed, serverless implementation of the Apache Iceberg REST Catalog API, letting Spark, Flink, Hive, Trino, and BigQuery share the same Iceberg tables without duplicating files.

Apache Gravitino is a federated metadata lake: rather than copying metadata into its own store, it manages metadata in place across Hive, JDBC databases, object storage, and message systems, exposing one model and one API for data and AI assets across regions and clouds.

Starburst is the commercial platform built on Trino, sold as Starburst Enterprise for self-managed clusters and Starburst Galaxy as a fully managed service, adding governance, performance work, and a supported Iceberg lakehouse on top of the open-source engine.

Dremio is a SQL engine for the lakehouse that federates queries across Iceberg, Delta Lake, databases, and warehouses, and accelerates them with materialised structures it calls Reflections — available as a free Community edition, a self-managed Enterprise edition, and a managed cloud service.

Daft is a Rust-based data engine with a Python interface, processing images, audio, video, embeddings, and structured data through one API, and scaling from a laptop to a cluster without code changes.

Apache Hudi is an open lakehouse format that pioneered transactional data lakes, offering record-level upserts and deletes, incremental queries, and a choice between copy-on-write and merge-on-read table types.

Apache Paimon is a lake format unifying batch and streaming, using an LSM tree to absorb high-frequency updates and emitting changelogs so downstream jobs can consume what changed rather than rescanning.

DuckLake is an integrated data lake and catalog format from the DuckDB Foundation that stores table metadata in an ACID SQL database rather than in files, with data in Parquet on object storage.

Lance is an open lakehouse format for multimodal AI data, claiming 100x faster random access than Parquet, with vector and full-text indexes in the format specification and versioning built in.

Google Cloud Storage is Google Cloud's managed object store, offering five storage classes from Rapid to Archive and an Autoclass mode that moves objects between them automatically as access patterns change.

Azure Blob Storage is Microsoft's managed object store, and the service Azure Data Lake Storage is built on — enabling its hierarchical namespace turns a blob container into a POSIX-like filesystem that Spark and Presto read through the ABFS driver.

Cloudflare R2 is a managed object store with an S3-compatible API and no charge for data transfer out at any volume, plus a built-in managed Apache Iceberg catalog currently in public beta.

Apache Ozone is an open-source object store built for analytics workloads, serving the same data through a native S3 protocol and a Hadoop-compatible filesystem interface, and scaling to billions of objects.

Databricks SQL is the warehousing surface of the Databricks platform, running BI and analytical SQL on the vectorized Photon engine against Delta Lake and Iceberg tables governed by Unity Catalog.

Microsoft Fabric Warehouse is the relational warehouse item in Microsoft Fabric, offering full T-SQL with multi-table ACID transactions over Delta tables held in OneLake and billed against a shared Fabric capacity.

StarRocks is an open-source MPP OLAP database combining streaming ingest, strong join performance at query time, and direct querying of lakehouse table formats.

Apache Iceberg is an open table format bringing ACID transactions, schema evolution, and time travel to data lake tables, with broad engine support.

Delta Lake is an open-source table format providing ACID transactions, schema enforcement, and time travel over Parquet files, closely integrated with Spark and Databricks.

Trino is an open-source distributed SQL engine that queries data where it lives — object storage, lakehouse table formats, and operational databases — through a single interface.

Amazon S3 is AWS's managed object storage service — the de facto substrate for data lakes and lakehouses, with tiered storage classes and fine-grained access control.

The AWS Glue Data Catalog is a fully managed, Hive-compatible metastore that resolves table definitions — including Iceberg tables — for engines across the AWS analytics stack.

Apache Polaris is an open-source catalog implementing the Iceberg REST specification, enabling interoperable, vendor-neutral table access across engines and clouds.

Apache Parquet is the standard columnar file format for analytics — compressed, splittable, and readable by effectively every engine, table format, and data tool.