Skip to content

Automatic translation — the Portuguese version is the reference. View in Portuguese

Updated on 2026-09-08

Tutorial — build the Bronze, Silver, and Gold layers

Step-by-step guide to the medallion — from raw data (Bronze) to clean data (Silver) to aggregated marts (Gold) with versioned transformations.

On this page (18)

Read first: Medallion pipeline · How data flows.


What it is#

The medallion organizes the datalake into three layers with clear roles:

  • Bronze — raw, verbatim data, as it arrived, auditable.
  • Silver — clean, typed, and deduplicated data, ready to model.
  • Gold — aggregated marts, ready for dashboards and Q&A.

What it is for#

Separating "what arrived" from "what is clean" from "what is ready" — so that a transformation error never destroys the original data and each layer has a single purpose.

When to use / when not to use#

  • Use it whenever you want reliable data for dashboards: the medallion is the default path for any source.
  • Do not jump straight to Gold without going through Silver — quality and deduplication live there.

Plan and permissions#

  • Core · access check · kill switches for ingestion runs and dashboard refresh.

Prerequisites#

  1. A connected source with data already in Bronze (via a full pipeline or import a file).

Step by step#

1) Bronze — raw and auditable

  • Ingestion writes each row verbatim (as text), guaranteeing the number of fields (fills missing ones with "") and adding the stamps _ingest_at, _source, _row.
  • Nothing is discarded in Bronze. It is your retention of the original data.
  • On the SaaS/landing path, raw files become NDJSON in landing/<prefix>/_saas/<sourceId>/<stamp>/<tabela>/chunk-NNNN.json and a single load job reads the chunk-*.json glob.

2) Silver — cleaning, typing, quality, and deduplication

Rules run row by row; one failure = the row does not enter Silver and is counted in rejectedRows. In the vendas example:

RuleBehavior
Status canceladodiscarded
Quantityinteger > 0
Priceaccepts Brazilian (1.299,90) and US (199.90) formats; > 0
Datevalid YYYY-MM-DD; derives mes = YYYY-MM
State (UF)2 uppercase letters
pedido_idrequired
Empty categorybecomes "Sem categoria" (not discarded)
Deduplicationby pedido_id, keeps the 1st occurrence
receitaderived = price × quantity

rejectedRows = bronzeRows − silverRows — the difference is always visible.

3) Silver → Gold transformations (versioned code)

Business rules are a versioned IR compiled to deterministic SQL. The available steps (named in Portuguese in the product): replace, text, dates, split, conditional, plus group-by/joins with grain-aware fan-out prevention. Versioning, parallel verification runs (running without publishing), and rollback are proven.

4) Gold — aggregated marts

Marts are rebuilt on every run (idempotent). In the example:

MartContents
gold_receita_mensalrevenue and orders by month
gold_receita_ufrevenue and orders by state (UF)
gold_receita_categoriarevenue and share by category
gold_kpistotal revenue, orders, average ticket, number of states, period

5) Publish

The Gold dashboard upsert republishes the dashboard automatically — running the pipeline makes the Gold data visible with no manual action.

Example (synthetic)#

A typical Silver → Gold business transformation:

code
1) datas: data_pedido → mes (YYYY-MM)
2) condicional: categoria vazia → "Sem categoria"
3) dividir: receita = preco_unitario × quantidade
4) group-by: SUM(receita), COUNT(*) por mes → gold_receita_mensal

Expected result#

  • Bronze with all the raw rows; Silver with the clean, deduplicated rows; Gold with the rebuilt marts.
  • A run record with Bronze / Silver / rejected / number of marts / duration / KPIs.

Limits and costs#

  • Profiling: cap of 50 columns, 5 samples × 60 chars, payload ≤ 32 KiB.
  • SaaS → Bronze load: default batchSize 5,000, maxRows 200,000.
  • Queries in the datalake are billable and capped by the billable-bytes ceiling.

Security#

  • Datasets scoped per workspace; column names interpolated only from an allowlist.
  • Schema drift warns when a column disappears / changes type (the mart does not break silently) — see Quality.

Common errors#

SymptomLikely causeWhat to do
Gold has less data than expectedmany rows rejected in SilverReview the rules that rejected them
Mart "broke" after a change in the sourceschema drift (atencao)Adjust the transformation to the new column/type
Different numbers after reprocessingexpected if the source changedBronze is verbatim; Silver/Gold reflect the current source

Troubleshooting#

  • Use the parallel verification run to validate a transformation before publishing.
  • Compare the Bronze/Silver/rejected counters across runs.

Related#


Last reviewed: 2026-09-08.

Related links