Skip to content

Automatic translation — the Portuguese version is the reference. View in Portuguese

Updated on 2026-09-08

Tutorial — create a full-load pipeline

Step-by-step guide to running a complete (full) pipeline that takes data from the source to the Gold marts and republishes the dashboard.

On this page (14)

Read first: How data flows · Medallion pipeline.


What it is#

A full pipeline recalculates everything from scratch on every run: it reads the source, loads Bronze verbatim, applies quality/deduplication in Silver, and rebuilds the Gold marts. It is the simplest and always correct mode — and the fallback for the incremental modes.

What it is for#

  • Doing the first load of a source (which already works as a backfill).
  • Guaranteeing maximum accuracy when the table is small or changes little.
  • Reprocessing everything after a business rule change.

When to use / when not to use#

  • Use it when: it is the 1st load; the table is small; the source deletes rows; or you want the strongest guarantee.
  • Do not use it when: the table is large and only grows — then incremental by watermark is much cheaper.

Plan and permissions#

  • Core · access check · kill switches for ingestion runs (load) and dashboard refresh (republishing the dashboard).

Prerequisites#

  1. A connected and tested source (e.g., PostgreSQL, BigQuery, or an imported file).
  2. At least one table selected for ingestion.

The two tracks#

The product shows progress live in serializable steps:

Ingesting a source (pipeline detail, Bronze tables section → Ingest live):

code
validar → landing → bronze → registrar

Engineering pipeline (pipeline page):

code
ingest → bronze → silver → gold → publish

Step by step#

  1. Open the pipeline, go to the Bronze tables section, and click Ingest live to bring the raw data into Bronze (validar → landing → bronze → registrar).
  2. Go to the pipeline page and run the complete track ingest → bronze → silver → gold → publish, leaving the mode set to full (default).
  3. On the run screen, follow the Bronze / Silver / rejected counters and the list of generated Gold marts.
  4. At the end, the Gold dashboard is republished automatically (you do not need to publish it by hand).

Example (synthetic)#

Source: a vendas table with the columns pedido_id, data_pedido, uf, categoria, preco_unitario, quantidade, status. After the full load, the typical Gold marts are:

MartContents
gold_receita_mensalrevenue and number of orders by month
gold_receita_ufrevenue and orders by state (UF)
gold_receita_categoriarevenue and share by category
gold_kpissingle row: total revenue, orders, average ticket, number of states, period

Expected result#

  • A run record with Bronze / Silver / rejected / number of marts / duration / KPIs.
  • rejectedRows = bronzeRows − silverRows — the difference is always visible.
  • The public Gold dashboard is updated and starts showing the newly processed data.

Limits and costs#

  • SaaS/file load → Bronze: default batchSize 5,000, maxRows 200,000.
  • A full load reads everything — it is the most expensive mode by definition. Queries in the datalake are billable and capped by the per-query read volume ceiling (default 1 GiB).

Security#

  • Datasets/buckets scoped per workspace (<prefix>_bronze/_silver/_gold).
  • Column names interpolated only from an allowlist; PII masked in profiling.

Common errors#

SymptomLikely causeWhat to do
Many rejected rowsdata outside the quality rulesSee which rules rejected them in Quality
Mart was not updatedpipeline stopped before gold/publishRun the complete track; check the run log
Load exceeded the bytes ceilingquery reads too muchNarrow the scope or adjust the ceiling deliberately

Troubleshooting#

  • Compare bronzeRows (what arrived) with silverRows (what passed cleaning).
  • Check the list of Gold marts on the run screen.

Related#


Last reviewed: 2026-09-08.

Related links