Build a Lakehouse

Warehouse-grade analytics on open table formats over object storage, without warehouse lock-in.

Last reviewed: Jul 30, 2026

Assumed context

Consumers

Analysts & BI

Internal analysts and BI users — not applications reading from the platform.

Freshness

Hourly, batch-first

An optional CDC path covers operational tables that need fresher data.

Scale

Terabytes

Raw data in the terabytes and growing; largest tables in the low billions of rows.

Team

2–6 engineers

No dedicated platform team — nobody's full-time job is running infrastructure.

Platform

Cloud, AWS

AWS as the concrete example; picks note portability where it matters.

Build vs buy

Managed where cheap

OSS-permissive for anything load-bearing, so the core stays swappable.

Optimizing for: Avoiding warehouse lock-in — table data that outlives any one engine.

Layers

Data flows left to right
Ingestion
Storage
Processing
Analytics
Cross-cutting layers

Batch ingestion

Batch ELT

Lands raw data from SaaS APIs, databases, and files into object storage on a schedule — append-only, schema captured at load time.

RecommendedFavicon of AirbyteAirbyte

Connector catalog covers the long tail of SaaS sources; the OSS deployment fits this team's self-host-cheap posture.

AlternativeFavicon of FivetranFivetran

Buy over build — if connector reliability is worth a per-row bill and nobody wants to operate ingestion.

Pipeline flow

↓ raw files

↓ change events

↓ tables

↓ modeled tables

↓ SQL

Cross-cutting layers