Skip to content

Automatic translation — the Portuguese version is the reference. View in Portuguese

Updated on 2026-09-08

Tutorial — create a quality validation

Step-by-step guide to applying quality rules in Silver, measuring rejected rows, and getting notified of schema changes.

On this page (14)

Read first: Quality, deduplication, and rejected rows · Medallion pipeline.


What it is#

A quality validation is the set of rules applied on the way from Bronze → Silver: whatever is valid moves on to Silver; whatever fails is counted and left out (the raw data stays in Bronze). This tutorial shows how to think through, apply, and measure these rules.

What it is for#

Keeping dirty data (invalid dates, zero prices, duplicates) from contaminating dashboards and answers — and measuring how much of your data is out of standard.

When to use / when not to use#

  • Use it always: every source that becomes a dashboard needs to go through Silver.
  • Do not treat Silver as a permanent discard — Bronze keeps the raw data for inspection and reprocessing.

Plan and permissions#

  • Core · access check · ingestion run kill switch.

Prerequisites#

  1. Data already in Bronze (via a full pipeline or import a file).
  2. Knowing the table's business key (to deduplicate).

Rule types (and what each one does)#

Rule typeEffect on the row
Required (e.g., pedido_id)empty → discards
Domain (e.g., status cancelado)forbidden value → discards
Type/format (date YYYY-MM-DD, integer > 0)invalid → discards
Normalization (state → 2 uppercase letters)adjusts; if it cannot, discards
Fill (empty category → "Sem categoria")fixes it, keeps the row
Derivation (receita = price × quantity)creates a calculated column
Deduplication (by key)keeps the 1st occurrence, discards the rest

Golden rule: fix when it is safe (fill/normalization) and discard when it is not (required/domain/type). Every discarded row counts toward the rejected total.

Step by step#

  1. Choose the deduplication key (e.g., pedido_id).
  2. List the hard rules (that discard): required fields, forbidden domains, date/number formats, valid state.
  3. List the soft rules (that fix): filling empty values, normalization.
  4. Define the derivations (e.g., receita).
  5. Apply them in the Silver transformation and run the pipeline.
  6. Measure: check rejectedRows = bronzeRows − silverRows in the run record.
  7. Investigate rejection spikes by looking at Bronze (the raw data is there).

Example (synthetic)#

Dirty data in Bronze:

csv
pedido_id;data_pedido;uf;categoria;preco_unitario;quantidade;status
P-1001;2026-01-03;SP;Bebidas;12,90;3;pago
P-1001;2026-01-03;SP;Bebidas;12,90;3;pago # duplicate → discards the 2nd
P-1002;2026/01/03;RJ;Limpeza;29,90;1;pago # invalid date → discards
P-1003;2026-01-04;Minas;Bebidas;12,90;2;pago # invalid state → discards
P-1004;2026-01-04;MG;;0;1;pago # price 0 → discards
P-1005;2026-01-05;MG;;12,90;2;cancelado # canceled → discards
P-1006;2026-01-05;SP;;12,90;2;pago # empty category → "Sem categoria" (kept)

Result: of 7 rows in Bronze, 1 passes to Silver (P-1006, with category "Sem categoria" and receita = 25,80); 6 are rejected — and you see exactly that in the counter.

Expected result#

  • Silver with only valid rows and no duplicates by key.
  • An explicit rejected counter per run (nothing disappears silently).
  • A schema drift warning when the source changes: new column → info; removed column / changed type / removed table → atencao (the mart that reads it is flagged as "breaks").

Limits and costs#

  • Safe profiling: cap of 50 columns, 5 samples × 60 chars, payload ≤ 32 KiB.
  • Rules run during the load (no extra platform cost beyond the load/queries).

Security#

  • PII column in profiling → samples •••, min/max/distinct suppressed.
  • Column hidden by governance → does not even appear.
  • Rejected raw data stays only in your workspace's Bronze, never exposed to another one.

Common errors#

SymptomLikely causeWhat to do
Very high rejection ratedate/price format different from expectedAdjust the rule (the price parser accepts Brazilian and US formats)
Duplicates in Goldwrong dedupe keyReview which column is the real business key
Mart stopped matchingschema drift atencaoAdjust the transformation to the new column/type

Troubleshooting#

  • Compare bronzeRows × silverRows on each run; a jump in rejections indicates a change in the source.
  • Use profiling to see the % of nulls and inferred × declared type (a mismatch = dirty data).

Related#


Last reviewed: 2026-09-08.

Related links