Dataform
Dataform is a service for developing, testing, version-controlling, and scheduling SQL transformation workflows in BigQuery, built on an Apache-2.0 meta-language and offered on Google Cloud at no charge beyond the BigQuery it runs.
LicenseOpen source (permissive)· Apache-2.0
DeploymentSelf-hostedManaged
PricingFree
Operational complexityLow
WorkloadBatch
LanguageTypeScript
Use it when
- The warehouse is BigQuery and will stay BigQuery; Dataform is Google's transformation service for it, free beyond the queries it issues.
- You want dbt-style modelling (a dependency graph, tests, docs, version control) with zero platform cost.
- Scheduling should stay inside Google Cloud, via workflow configurations, Cloud Scheduler, or Composer.
- Git-backed development with GitHub, GitLab, Azure DevOps, or Bitbucket is required.
Think twice when
- A second engine is ever plausible; BigQuery is the only target, where dbt and SQLMesh are portable.
- Column-level lineage or a semantic layer is needed; the graph is action-level and there is no metrics layer.
- The team dislikes the JavaScript meta-language compared with dbt's Jinja or SQLMesh's plain SQL.
- Incremental logic needs guardrails; nothing classifies a change, you write the conditions yourself.
How it runs
A managed Google Cloud service: development workspaces edit a Git-backed repository, release configurations compile with per-environment overrides, and workflow configurations execute on unix-cron schedules. The service is free; you pay for the BigQuery queries it runs plus required Cloud Logging. Dataform Core is Apache-2.0 and runnable locally through its CLI.
Details
CompareHow Dataform answers the questions Transformation turns on.
Transformation
| How it works | |
| Model language | SQL in a JavaScript meta-language (Dataform Core) |
| Model kinds | Table, incremental table, view and materialized view; declarations name external sources, operations run arbitrary SQL, assertions test it |
| Change detection | None: an incremental table is guarded by your own when(incremental()) condition, and a full refresh rebuilds it; nothing classifies an edit or decides what to rebuild |
| Testing | Assertions only: built-in nonNull, uniqueKey and rowConditions, or custom SQL that has to return zero rows; results land in their own schema |
| Lineage | An interactive compiled dependency graph in the console, at action rather than column level |
| Semantic layer | None |
| Running it | |
| Environments | Development workspaces are editable copies of the repository, and release configurations compile with overrides so dev and prod write to different projects or schemas; nothing is shared, each environment builds its own tables |
| Scheduling | Release then workflow configurations on a unix-cron schedule, built in; Workflows with Cloud Scheduler, Cloud Composer or a Cloud Build trigger on commit are the alternatives |
| Connections | |
| Engines | BigQuery only |
| Cost | |
| Billing unit | The service itself is free, you pay BigQuery for every query it runs, plus Cloud Logging, which is required and on by default |
Alternatives to Dataform
