Storage & Table Formats

Apache Hudi is an open lakehouse format that pioneered transactional data lakes, offering record-level upserts and deletes, incremental queries, and a choice between copy-on-write and merge-on-read table types.

Apache Paimon is a lake format unifying batch and streaming, using an LSM tree to absorb high-frequency updates and emitting changelogs so downstream jobs can consume what changed rather than rescanning.

DuckLake is an integrated data lake and catalog format from the DuckDB Foundation that stores table metadata in an ACID SQL database rather than in files, with data in Parquet on object storage.

Lance is an open lakehouse format for multimodal AI data, claiming 100x faster random access than Parquet, with vector and full-text indexes in the format specification and versioning built in.

Apache Druid is a real-time analytics database combining ideas from data warehouses, time-series databases, and log search systems, with columnar storage, time-based partitioning, and bitmap indexes serving sub-second queries over streaming data.

Apache Doris is an MPP analytical database for real-time reporting and ad-hoc analysis, speaking MySQL-compatible SQL over a Java front end and a C++ back end, and the project StarRocks was originally forked from.

SingleStore is a commercial distributed SQL database that runs transactional and analytical workloads in one engine, combining rowstore and columnstore tables with vector search, and delivered as the managed Helios service or self-managed.

Tinybird is a managed ClickHouse platform that turns a SQL query into a deployed HTTP endpoint, aimed at developers building real-time analytics into applications rather than operating a database.

TimescaleDB is a PostgreSQL extension that adds automatic time-based partitioning, hybrid row-columnar storage, and incrementally refreshed aggregates, keeping full PostgreSQL SQL and its entire ecosystem of drivers and tools.

QuestDB is an open-source time-series database with a column-oriented, time-partitioned storage engine and SIMD-vectorized execution, extending SQL with time-series operators like SAMPLE BY and ASOF JOIN.

kdb+ is KX's commercial columnar time-series database, long the standard for financial tick data, queried through the vector language q rather than standard SQL.

CrateDB is an open-source distributed SQL database for real-time analytics and search, querying relational, JSON, time-series, geospatial, vector, and full-text data through one PostgreSQL-compatible interface.

GreptimeDB is an open-source observability database handling metrics, logs, and traces in one Rust engine, with columnar storage on object storage and both SQL and PromQL as query languages.

Google Cloud Storage is Google Cloud's managed object store, offering five storage classes from Rapid to Archive and an Autoclass mode that moves objects between them automatically as access patterns change.

Azure Blob Storage is Microsoft's managed object store, and the service Azure Data Lake Storage is built on — enabling its hierarchical namespace turns a blob container into a POSIX-like filesystem that Spark and Presto read through the ABFS driver.

Cloudflare R2 is a managed object store with an S3-compatible API and no charge for data transfer out at any volume, plus a built-in managed Apache Iceberg catalog currently in public beta.

Ceph is an open-source distributed storage platform providing object, block, and file interfaces from a single cluster built on commodity hardware, with its RADOS Gateway exposing the S3-compatible API.

Apache Ozone is an open-source object store built for analytics workloads, serving the same data through a native S3 protocol and a Hadoop-compatible filesystem interface, and scaling to billions of objects.

SeaweedFS is an open-source distributed store built on Facebook's Haystack design, keeping per-file metadata overhead to about 40 bytes so it can serve billions of small files with a single disk seek.

Garage is an open-source S3-compatible object store that ships as one dependency-free binary, runs across geographically separate sites without a dedicated backbone, and needs as little as 1 GB of RAM per node.

Backblaze B2 is a managed S3-compatible object store priced well below the hyperscalers, with free monthly egress up to three times the stored volume and no minimum storage duration.

Wasabi is a managed S3-compatible object store billed at a flat rate per terabyte with no charges for egress or API operations, in exchange for a 90-day minimum storage duration.

Storj is an S3-compatible object store that erasure-codes each object across a global network of independently operated nodes, rather than storing it in a provider's own datacenters.

Amazon Redshift is AWS's managed analytical data warehouse, offering provisioned RA3 clusters with managed storage or a serverless mode that scales compute automatically and bills per second.

Databricks SQL is the warehousing surface of the Databricks platform, running BI and analytical SQL on the vectorized Photon engine against Delta Lake and Iceberg tables governed by Unity Catalog.

Microsoft Fabric Warehouse is the relational warehouse item in Microsoft Fabric, offering full T-SQL with multi-table ACID transactions over Delta tables held in OneLake and billed against a shared Fabric capacity.

Oracle Autonomous AI Database is a fully managed analytical database on Exadata infrastructure that automates provisioning, tuning, patching, and backup, with an Always Free tier and independently elastic compute and storage.

Firebolt is a cloud data warehouse built for sub-second analytics on customer-facing applications, pairing stateless elastic engines with primary, join, and aggregating indexes, and shipping a free self-hosted edition alongside the managed service.

MotherDuck is a serverless cloud data warehouse built on DuckDB, whose dual execution engine splits a single SQL query between the developer's laptop and the cloud depending on where the data lives.

Apache Cloudberry is an open-source massively parallel processing data warehouse built on a PostgreSQL 14 kernel, created by original Greenplum developers after Greenplum itself was closed-sourced.

Teradata VantageCloud is the cloud form of Teradata's massively parallel analytical platform, offering elastic object-store-backed compute in VantageCloud Lake and the established Vantage architecture in VantageCloud Enterprise.

Vertica is a columnar massively parallel analytics database that runs on-premises or in any cloud, in Enterprise mode with local storage or Eon mode with communal object storage and independently scalable compute.

Exasol is an in-memory, columnar, massively parallel analytics database available as a managed SaaS on S3-backed storage, deployed into a customer's own cloud account, or run on-premises.

Apache Pinot is a distributed real-time OLAP store built to serve analytical queries at very high concurrency with millisecond latency, ingesting directly from streams.

StarRocks is an open-source MPP OLAP database combining streaming ingest, strong join performance at query time, and direct querying of lakehouse table formats.

Snowflake is a commercial cloud data platform offering elastic SQL warehouses, cross-account data sharing, and multi-cloud availability as a fully managed service.