Read first: How data flows · Medallion pipeline.
What it is#
A full pipeline recalculates everything from scratch on every run: it reads the source, loads Bronze verbatim, applies quality/deduplication in Silver, and rebuilds the Gold marts. It is the simplest and always correct mode — and the fallback for the incremental modes.
What it is for#
- Doing the first load of a source (which already works as a backfill).
- Guaranteeing maximum accuracy when the table is small or changes little.
- Reprocessing everything after a business rule change.
When to use / when not to use#
- Use it when: it is the 1st load; the table is small; the source deletes rows; or you want the strongest guarantee.
- Do not use it when: the table is large and only grows — then incremental by watermark is much cheaper.
Plan and permissions#
- Core · access check · kill switches for ingestion runs (load) and dashboard refresh (republishing the dashboard).
Prerequisites#
- A connected and tested source (e.g., PostgreSQL, BigQuery, or an imported file).
- At least one table selected for ingestion.
The two tracks#
The product shows progress live in serializable steps:
Ingesting a source (pipeline detail, Bronze tables section → Ingest live):
validar → landing → bronze → registrarEngineering pipeline (pipeline page):
ingest → bronze → silver → gold → publishStep by step#
- Open the pipeline, go to the Bronze tables section, and click Ingest live to bring the raw data into
Bronze (
validar → landing → bronze → registrar). - Go to the pipeline page and run the complete track
ingest → bronze → silver → gold → publish, leaving the mode set to full (default). - On the run screen, follow the Bronze / Silver / rejected counters and the list of generated Gold marts.
- At the end, the Gold dashboard is republished automatically (you do not need to publish it by hand).
Example (synthetic)#
Source: a vendas table with the columns pedido_id, data_pedido, uf, categoria, preco_unitario, quantidade, status. After the full load, the typical Gold marts are:
| Mart | Contents |
|---|---|
gold_receita_mensal | revenue and number of orders by month |
gold_receita_uf | revenue and orders by state (UF) |
gold_receita_categoria | revenue and share by category |
gold_kpis | single row: total revenue, orders, average ticket, number of states, period |
Expected result#
- A run record with Bronze / Silver / rejected / number of marts / duration / KPIs.
rejectedRows = bronzeRows − silverRows— the difference is always visible.- The public Gold dashboard is updated and starts showing the newly processed data.
Limits and costs#
- SaaS/file load → Bronze: default
batchSize5,000,maxRows200,000. - A full load reads everything — it is the most expensive mode by definition. Queries in the datalake are billable and capped by the per-query read volume ceiling (default 1 GiB).
Security#
- Datasets/buckets scoped per workspace (
<prefix>_bronze/_silver/_gold). - Column names interpolated only from an allowlist; PII masked in profiling.
Common errors#
| Symptom | Likely cause | What to do |
|---|---|---|
| Many rejected rows | data outside the quality rules | See which rules rejected them in Quality |
| Mart was not updated | pipeline stopped before gold/publish | Run the complete track; check the run log |
| Load exceeded the bytes ceiling | query reads too much | Narrow the scope or adjust the ceiling deliberately |
Troubleshooting#
- Compare
bronzeRows(what arrived) withsilverRows(what passed cleaning). - Check the list of Gold marts on the run screen.
Related#
Last reviewed: 2026-09-08.