Skip to content

Automatic translation — the Portuguese version is the reference. View in Portuguese

Updated on 2026-09-08

Data Tutorials — step by step

Index of the practical tutorials for the Data layer — connect sources, run pipelines, backfill, quality and failure handling.

On this page (5)

All examples are synthetic (.example domains, fictitious credentials).

Read before: How the data flows (overview). The conceptual and state reference lives on the data/ pages; here are the step-by-step guides.


Connect a source#

TutorialCovers
Connect PostgreSQLRelational database by pull, connection test, SSL, network (allowlist/VPN/SSH)
Connect client cloud analytics datasetClient's cloud analytics dataset as a source, data project × billing, service account
Import a filePush of CSV/JSON/Parquet to the landing and load into Bronze
Connect a generic REST APISystem without a ready connector: URL, list path, pagination (page/offset/URL), Bearer token, discovered columns

Run the pipeline#

TutorialCovers
Create full pipelineFull load source → Bronze → Silver → Gold → publish
Create incremental pipeline (watermark)Delta by watermark (KEYSET/upsert); not CDC
Run backfillHistorical data prior to the window; one-off full, wide-window, checkpoint
Build Bronze/Silver/GoldThe medallion layer, layer by layer, with versioned transformations
Add a column to a Silver or Gold tableBring a new data point (e.g., clicks) into an existing table: edit the recipe, run and update the dashboard

Quality and failures#

TutorialCovers
Create quality validationRules on Silver, deduplication, rejected records, schema drift
Handle failure and DLQRetry with backoff, DLQ, reconciliation across layers

Suggested sequence for those starting#

  1. Connect a source — start with a database, the client cloud analytics dataset or a file.
  2. Run a full pipeline to bring the data up to Gold.
  3. Validate quality and adjust the rules.
  4. Optimize with incremental by watermark and bring history with backfill.
  5. Prepare for failures with the DLQ guide.

Conventions of these tutorials#

  • Honesty first: nothing here promises nonexistent features. Incremental load is by watermark (CDC is Roadmap). Amazon S3, GCS and Azure Blob Storage are in Limited access (each connector page states the condition).
  • Synthetic examples: hosts *.example, fictitious projects/credentials, zero PII.

Last review: 2026-09-08.

Related links