Technologies tagged with "Python"

Soda is a data quality platform built on Soda Core, a Python library and CLI that runs checks and data contracts inside pipelines, with Soda Cloud for triage and collaboration and a hosted or self-hosted Runner for scheduled observability.

Elementary is an Apache-2.0 data observability tool built for dbt projects: a dbt package and CLI collect test results and run anomaly detection on freshness, volume, and custom metrics, with Elementary Cloud adding column-level lineage, ML monitors, and incident workflows.

pandera is an MIT-licensed Python library for validating dataframes: schemas are declared as objects or as typed classes and checked at runtime across pandas, polars, pyspark, dask, modin, ibis, and geopandas.

Flyte is an LF AI & Data project for building production data and ML pipelines: tasks and workflows are Python functions with strongly typed interfaces, each execution is versioned and cacheable, and the whole thing runs on Kubernetes with a Go control plane underneath.

Luigi is Spotify's Python library for building pipelines of batch jobs: tasks declare what they require and what they produce, a central scheduler resolves the graph and prevents duplicate work, and triggering is left to cron.

Redash is a BSD-licensed query-and-dashboard tool: analysts write SQL against any connected source, turn results into visualisations, assemble them into dashboards, and share them — running entirely on the team's own infrastructure, with no commercial edition to upgrade to.

SQLMesh is a data transformation framework from Tobiko Data that runs SQL and Python models with a plan-and-apply workflow, column-level lineage, unit tests, and virtual data environments that let teams preview changes without rebuilding tables.

Bruin is an open-source data pipeline tool written in Go that combines ingestion, SQL and Python transformations, automated quality checks, and column-level lineage in a single CLI, with a managed cloud layer above it.

Apache DataFusion is an extensible query engine written in Rust that uses Apache Arrow as its in-memory format, shipped as libraries for building database and analytic systems rather than as a server to run.

Polars is a DataFrame library written in Rust with a Python interface, built around a query optimiser and Arrow memory rather than row-by-row execution, and designed as a single-node engine that handles data most teams would otherwise put on a cluster.

Ray is an open-source framework for scaling Python and AI workloads, pairing a general-purpose distributed execution core with libraries for data, training, tuning, serving, and reinforcement learning.

Dask is a Python library for parallel and distributed computing that provides pandas-compatible DataFrames and NumPy-compatible Arrays over datasets larger than memory, running locally or across a cluster.

Daft is a Rust-based data engine with a Python interface, processing images, audio, video, embeddings, and structured data through one API, and scaling from a laptop to a cluster without code changes.

Apache Beam is a unified programming model for batch and streaming pipelines, written once in Java, Python, or Go and executed by a runner — Flink, Spark, Dataflow, or others — rather than by an engine of its own.

dlt is an open-source Python library for the extract-and-load stage, inferring schemas, normalising nested data, and handling incremental loading — installed with pip and run wherever Python already runs.

Meltano is an open-source ELT orchestrator that assembles pipelines from Singer taps and targets declaratively in version-controlled YAML, with a hub cataloguing over 600 community-maintained connectors.

Karapace is an Apache-2.0 drop-in replacement for Confluent Schema Registry (plus a Kafka REST proxy), maintained by Aiven.

Apache Spark is an open-source distributed compute engine for batch processing, SQL, machine learning, and stream processing across large clusters.

DuckDB is an open-source, in-process OLAP database — an embeddable columnar SQL engine that queries Parquet, CSV, and dataframes with no server to run.

Apache Airflow is the widely adopted open-source orchestrator where pipelines are Python-defined DAGs, with a large provider ecosystem and multiple managed offerings.

Dagster is an open-source orchestrator organized around software-defined assets — declaring the tables and models pipelines produce — with strong local development and testing.

Prefect is an open-source orchestration framework where plain Python functions become flows and tasks, aimed at dynamic, failure-tolerant pipelines with minimal boilerplate.

Great Expectations (GX) is an open-source Python framework for declaring, validating, and documenting data quality expectations inside pipelines.

Apache Superset is an open-source BI platform with a rich visualization library, SQL IDE, and fine-grained access control, built for customization and larger deployments.