Read first: Quality, deduplication, and rejected rows · Medallion pipeline.
What it is#
A quality validation is the set of rules applied on the way from Bronze → Silver: whatever is valid moves on to Silver; whatever fails is counted and left out (the raw data stays in Bronze). This tutorial shows how to think through, apply, and measure these rules.
What it is for#
Keeping dirty data (invalid dates, zero prices, duplicates) from contaminating dashboards and answers — and measuring how much of your data is out of standard.
When to use / when not to use#
- Use it always: every source that becomes a dashboard needs to go through Silver.
- Do not treat Silver as a permanent discard — Bronze keeps the raw data for inspection and reprocessing.
Plan and permissions#
- Core · access check · ingestion run kill switch.
Prerequisites#
- Data already in Bronze (via a full pipeline or import a file).
- Knowing the table's business key (to deduplicate).
Rule types (and what each one does)#
| Rule type | Effect on the row |
|---|---|
Required (e.g., pedido_id) | empty → discards |
Domain (e.g., status cancelado) | forbidden value → discards |
Type/format (date YYYY-MM-DD, integer > 0) | invalid → discards |
| Normalization (state → 2 uppercase letters) | adjusts; if it cannot, discards |
Fill (empty category → "Sem categoria") | fixes it, keeps the row |
Derivation (receita = price × quantity) | creates a calculated column |
| Deduplication (by key) | keeps the 1st occurrence, discards the rest |
Golden rule: fix when it is safe (fill/normalization) and discard when it is not (required/domain/type). Every discarded row counts toward the rejected total.
Step by step#
- Choose the deduplication key (e.g.,
pedido_id). - List the hard rules (that discard): required fields, forbidden domains, date/number formats, valid state.
- List the soft rules (that fix): filling empty values, normalization.
- Define the derivations (e.g.,
receita). - Apply them in the Silver transformation and run the pipeline.
- Measure: check
rejectedRows = bronzeRows − silverRowsin the run record. - Investigate rejection spikes by looking at Bronze (the raw data is there).
Example (synthetic)#
Dirty data in Bronze:
pedido_id;data_pedido;uf;categoria;preco_unitario;quantidade;status
P-1001;2026-01-03;SP;Bebidas;12,90;3;pago
P-1001;2026-01-03;SP;Bebidas;12,90;3;pago # duplicate → discards the 2nd
P-1002;2026/01/03;RJ;Limpeza;29,90;1;pago # invalid date → discards
P-1003;2026-01-04;Minas;Bebidas;12,90;2;pago # invalid state → discards
P-1004;2026-01-04;MG;;0;1;pago # price 0 → discards
P-1005;2026-01-05;MG;;12,90;2;cancelado # canceled → discards
P-1006;2026-01-05;SP;;12,90;2;pago # empty category → "Sem categoria" (kept)Result: of 7 rows in Bronze, 1 passes to Silver (P-1006, with category
"Sem categoria" and receita = 25,80); 6 are rejected — and you see exactly that in the
counter.
Expected result#
- Silver with only valid rows and no duplicates by key.
- An explicit rejected counter per run (nothing disappears silently).
- A schema drift warning when the source changes: new column →
info; removed column / changed type / removed table →atencao(the mart that reads it is flagged as "breaks").
Limits and costs#
- Safe profiling: cap of 50 columns, 5 samples × 60 chars, payload ≤ 32 KiB.
- Rules run during the load (no extra platform cost beyond the load/queries).
Security#
- PII column in profiling → samples
•••, min/max/distinct suppressed. - Column hidden by governance → does not even appear.
- Rejected raw data stays only in your workspace's Bronze, never exposed to another one.
Common errors#
| Symptom | Likely cause | What to do |
|---|---|---|
| Very high rejection rate | date/price format different from expected | Adjust the rule (the price parser accepts Brazilian and US formats) |
| Duplicates in Gold | wrong dedupe key | Review which column is the real business key |
| Mart stopped matching | schema drift atencao | Adjust the transformation to the new column/type |
Troubleshooting#
- Compare
bronzeRows×silverRowson each run; a jump in rejections indicates a change in the source. - Use profiling to see the % of nulls and inferred × declared type (a mismatch = dirty data).
Related#
- Quality, deduplication, and rejected rows (concept)
- Build Bronze/Silver/Gold
- Handle failures and the DLQ
Last reviewed: 2026-09-08.