All examples are synthetic (
.exampledomains, fictitious credentials).
Read before: How the data flows (overview). The conceptual and state reference lives on the
data/pages; here are the step-by-step guides.
Connect a source#
| Tutorial | Covers |
|---|---|
| Connect PostgreSQL | Relational database by pull, connection test, SSL, network (allowlist/VPN/SSH) |
| Connect client cloud analytics dataset | Client's cloud analytics dataset as a source, data project × billing, service account |
| Import a file | Push of CSV/JSON/Parquet to the landing and load into Bronze |
| Connect a generic REST API | System without a ready connector: URL, list path, pagination (page/offset/URL), Bearer token, discovered columns |
Run the pipeline#
| Tutorial | Covers |
|---|---|
| Create full pipeline | Full load source → Bronze → Silver → Gold → publish |
| Create incremental pipeline (watermark) | Delta by watermark (KEYSET/upsert); not CDC |
| Run backfill | Historical data prior to the window; one-off full, wide-window, checkpoint |
| Build Bronze/Silver/Gold | The medallion layer, layer by layer, with versioned transformations |
| Add a column to a Silver or Gold table | Bring a new data point (e.g., clicks) into an existing table: edit the recipe, run and update the dashboard |
Quality and failures#
| Tutorial | Covers |
|---|---|
| Create quality validation | Rules on Silver, deduplication, rejected records, schema drift |
| Handle failure and DLQ | Retry with backoff, DLQ, reconciliation across layers |
Suggested sequence for those starting#
- Connect a source — start with a database, the client cloud analytics dataset or a file.
- Run a full pipeline to bring the data up to Gold.
- Validate quality and adjust the rules.
- Optimize with incremental by watermark and bring history with backfill.
- Prepare for failures with the DLQ guide.
Conventions of these tutorials#
- Honesty first: nothing here promises nonexistent features. Incremental load is by watermark (CDC is Roadmap). Amazon S3, GCS and Azure Blob Storage are in Limited access (each connector page states the condition).
- Synthetic examples: hosts
*.example, fictitious projects/credentials, zero PII.
Last review: 2026-09-08.