Technologies tagged with "Jvm"

Apicurio Registry is an Apache-2.0 registry that stores schemas and API definitions together — Avro, Protobuf, JSON Schema, OpenAPI, AsyncAPI, GraphQL, WSDL and XSD — behind a REST API, with a Confluent-compatible endpoint so existing Kafka clients work unchanged.

Deequ is an Apache-2.0 library from AWS Labs for defining "unit tests for data" on Apache Spark: constraints and metrics are computed as part of the Spark job itself, which makes quality checks scale with the data rather than sampling it.

Apache Atlas is an Apache Software Foundation project providing metadata management, classification, and lineage for the Hadoop ecosystem, with an extensible type system and an authorization path through Apache Ranger.

Apache Gravitino is a federated metadata lake: rather than copying metadata into its own store, it manages metadata in place across Hive, JDBC databases, object storage, and message systems, exposing one model and one API for data and AI assets across regions and clouds.

Kestra is an open-source orchestration platform whose workflows are declared in YAML rather than written in a programming language: an event-driven, language-agnostic engine with a large plugin catalogue, a JVM runtime backed by Postgres or MySQL, and Enterprise and Cloud editions above the Apache-2.0 core.

Apache DolphinScheduler is a distributed workflow orchestration platform with a drag-and-drop DAG editor, a decentralised multi-master and multi-worker architecture built for high availability, and built-in task types for the tools a data platform already runs.

Presto is a distributed SQL query engine for federated analytics, governed by the Presto Foundation under the Linux Foundation, whose members include Meta, IBM, Uber, and Nvidia — the original codebase that PrestoSQL forked from before renaming itself Trino.

Dremio is a SQL engine for the lakehouse that federates queries across Iceberg, Delta Lake, databases, and warehouses, and accelerates them with materialised structures it calls Reflections — available as a free Community edition, a self-managed Enterprise edition, and a managed cloud service.

Apache Hive is a distributed, fault-tolerant data warehouse system that enables analytics at massive scale, running SQL over data in the Hadoop ecosystem on Apache Tez, with LLAP for sub-second queries and first-class Apache Iceberg support in the 4.x line.

Apache Hadoop is the distributed storage and batch processing framework that started the category, comprising HDFS for storage, YARN for resource management and scheduling, and MapReduce as its batch execution engine.

Apache Beam is a unified programming model for batch and streaming pipelines, written once in Java, Python, or Go and executed by a runner — Flink, Spark, Dataflow, or others — rather than by an engine of its own.

Apache Pulsar is an open-source messaging and streaming platform that splits brokers from storage, keeping segments in Apache BookKeeper so capacity and serving scale independently, with multi-tenancy and geo-replication built in.

AutoMQ keeps Apache Kafka's compute layer and replaces its storage with cloud object storage, claiming 100% Kafka API compatibility and substantially lower cost by removing multi-replica local disks and cross-zone transfer.

Maxwell's Daemon reads the MySQL binary log and writes row-level changes as JSON to Kafka, Kinesis, and other streaming platforms — a single-purpose CDC process with no framework around it.

Apache Druid is a real-time analytics database combining ideas from data warehouses, time-series databases, and log search systems, with columnar storage, time-based partitioning, and bitmap indexes serving sub-second queries over streaming data.

Apache Doris is an MPP analytical database for real-time reporting and ad-hoc analysis, speaking MySQL-compatible SQL over a Java front end and a C++ back end, and the project StarRocks was originally forked from.

QuestDB is an open-source time-series database with a column-oriented, time-partitioned storage engine and SIMD-vectorized execution, extending SQL with time-series operators like SAMPLE BY and ASOF JOIN.

CrateDB is an open-source distributed SQL database for real-time analytics and search, querying relational, JSON, time-series, geospatial, vector, and full-text data through one PostgreSQL-compatible interface.

Apache Ozone is an open-source object store built for analytics workloads, serving the same data through a native S3 protocol and a Hadoop-compatible filesystem interface, and scaling to billions of objects.

Apache Pinot is a distributed real-time OLAP store built to serve analytical queries at very high concurrency with millisecond latency, ingesting directly from streams.

Confluent Schema Registry stores versioned Avro, Protobuf, and JSON schemas for Kafka topics and enforces compatibility rules when producers register changes.

Debezium is an open-source CDC platform that streams row-level changes from databases such as PostgreSQL, MySQL, and MongoDB, typically via Kafka Connect.

Apache Spark is an open-source distributed compute engine for batch processing, SQL, machine learning, and stream processing across large clusters.

Trino is an open-source distributed SQL engine that queries data where it lives — object storage, lakehouse table formats, and operational databases — through a single interface.

Apache Kafka is an open-source distributed log for high-throughput, durable event streaming, with a large ecosystem including Kafka Connect and Kafka Streams.

Apache Flink is an open-source framework for stateful computation over unbounded and bounded data streams, with event-time processing, exactly-once state consistency, and CDC ingestion via Flink CDC.