Read first: Medallion pipeline · How data flows.
What it is#
The medallion organizes the datalake into three layers with clear roles:
- Bronze — raw, verbatim data, as it arrived, auditable.
- Silver — clean, typed, and deduplicated data, ready to model.
- Gold — aggregated marts, ready for dashboards and Q&A.
What it is for#
Separating "what arrived" from "what is clean" from "what is ready" — so that a transformation error never destroys the original data and each layer has a single purpose.
When to use / when not to use#
- Use it whenever you want reliable data for dashboards: the medallion is the default path for any source.
- Do not jump straight to Gold without going through Silver — quality and deduplication live there.
Plan and permissions#
- Core · access check · kill switches for ingestion runs and dashboard refresh.
Prerequisites#
- A connected source with data already in Bronze (via a full pipeline or import a file).
Step by step#
1) Bronze — raw and auditable
- Ingestion writes each row verbatim (as text), guaranteeing the number of fields
(fills missing ones with
"") and adding the stamps_ingest_at,_source,_row. - Nothing is discarded in Bronze. It is your retention of the original data.
- On the SaaS/landing path, raw files become NDJSON in
landing/<prefix>/_saas/<sourceId>/<stamp>/<tabela>/chunk-NNNN.jsonand a single load job reads thechunk-*.jsonglob.
2) Silver — cleaning, typing, quality, and deduplication
Rules run row by row; one failure = the row does not enter Silver and is
counted in rejectedRows. In the vendas example:
| Rule | Behavior |
|---|---|
Status cancelado | discarded |
| Quantity | integer > 0 |
| Price | accepts Brazilian (1.299,90) and US (199.90) formats; > 0 |
| Date | valid YYYY-MM-DD; derives mes = YYYY-MM |
| State (UF) | 2 uppercase letters |
pedido_id | required |
| Empty category | becomes "Sem categoria" (not discarded) |
| Deduplication | by pedido_id, keeps the 1st occurrence |
receita | derived = price × quantity |
rejectedRows = bronzeRows − silverRows — the difference is always visible.
3) Silver → Gold transformations (versioned code)
Business rules are a versioned IR compiled to deterministic SQL. The available steps (named in Portuguese in the product): replace, text, dates, split, conditional, plus group-by/joins with grain-aware fan-out prevention. Versioning, parallel verification runs (running without publishing), and rollback are proven.
4) Gold — aggregated marts
Marts are rebuilt on every run (idempotent). In the example:
| Mart | Contents |
|---|---|
gold_receita_mensal | revenue and orders by month |
gold_receita_uf | revenue and orders by state (UF) |
gold_receita_categoria | revenue and share by category |
gold_kpis | total revenue, orders, average ticket, number of states, period |
5) Publish
The Gold dashboard upsert republishes the dashboard automatically — running the pipeline makes the Gold data visible with no manual action.
Example (synthetic)#
A typical Silver → Gold business transformation:
1) datas: data_pedido → mes (YYYY-MM)
2) condicional: categoria vazia → "Sem categoria"
3) dividir: receita = preco_unitario × quantidade
4) group-by: SUM(receita), COUNT(*) por mes → gold_receita_mensalExpected result#
- Bronze with all the raw rows; Silver with the clean, deduplicated rows; Gold with the rebuilt marts.
- A run record with Bronze / Silver / rejected / number of marts / duration / KPIs.
Limits and costs#
- Profiling: cap of 50 columns, 5 samples × 60 chars, payload ≤ 32 KiB.
- SaaS → Bronze load: default
batchSize5,000,maxRows200,000. - Queries in the datalake are billable and capped by the billable-bytes ceiling.
Security#
- Datasets scoped per workspace; column names interpolated only from an allowlist.
- Schema drift warns when a column disappears / changes type (the mart does not break silently) — see Quality.
Common errors#
| Symptom | Likely cause | What to do |
|---|---|---|
| Gold has less data than expected | many rows rejected in Silver | Review the rules that rejected them |
| Mart "broke" after a change in the source | schema drift (atencao) | Adjust the transformation to the new column/type |
| Different numbers after reprocessing | expected if the source changed | Bronze is verbatim; Silver/Gold reflect the current source |
Troubleshooting#
- Use the parallel verification run to validate a transformation before publishing.
- Compare the Bronze/Silver/rejected counters across runs.
Related#
- Medallion pipeline (reference)
- Create a quality validation
- Create a full pipeline
- Add a column to a Silver or Gold table
Last reviewed: 2026-09-08.