Read first:
conceitos-fluxo-de-dados.md. Silver layer reference inpipeline-medalhao.md.
The principle: raw data is never lost#
Ingestia's quality checks are applied on the Bronze → Silver step. The core guarantee is that Bronze is immutable and verbatim: each source row enters as text, with no conversion, with audit stamps. Therefore, no row disappears — even a row rejected during cleaning still exists in Bronze, available for inspection and reprocessing.
Bronze (everything, raw) → quality rules + dedupe → Silver (only valid, clean)
│
└──▶ rejected = Bronze − Silver (counted, visible)The quality rules (reference example)#
The reference implementation (synthetic vendas table) applies the rules row by
row. A failure = the row does not enter Silver and is counted in rejectedRows:
| Rule | Behavior |
|---|---|
Status cancelado | discarded |
| Quantity | integer > 0, otherwise discarded |
| Price | accepts BR (1.299,90) and US (199.90) formats; > 0, otherwise discarded |
| Date | valid YYYY-MM-DD, otherwise discarded; derives mes = YYYY-MM |
| State (UF) | normalized to 2 uppercase letters, otherwise discarded |
pedido_id | required |
| Empty category | becomes "Sem categoria" (not discarded) |
| Deduplication | by pedido_id, keeps the 1st occurrence |
receita | derived = price × quantity |
Note the difference between discarding (the row failed a hard rule) and fixing (empty category becomes "Sem categoria" — the row is kept). The product fixes when it is safe and discards when it is not.
Rejected rows: the difference is always visible#
The number of rejected rows is always explicit:
rejectedRows = bronzeRows − silverRowsEach run records this trio (Bronze / Silver / rejected) in the run history. You see, on every run, how many rows came in raw, how many survived cleaning and how many were left out. Nothing is discarded silently.
Deduplication#
Deduplication happens in Silver, by business key (in the example, pedido_id),
keeping the first occurrence. Subsequent duplicates of the same key are
discarded (and count toward the rejected total). This makes Silver idempotent by
key, even if the source sends the same row more than once.
"Quarantine": what the product actually does (honesty)#
There is no separate quarantine table today for rows rejected in the Silver layer. What exists — and plays the same role — is the combination of:
- Verbatim Bronze keeps all raw rows (the "retention" of what arrived);
- the rejected counter (
bronzeRows − silverRows) makes the loss measurable; - automatic load sanitization preserves problematic file rows instead of inventing data (an unparseable row is kept as is).
In other words: you do not lose the rejected data (it is in Bronze) and you know how
much was rejected (counter), but Ingestia does not promise today a dedicated
quarantine queue with a per-row fix/resend flow. Do not confuse this with the
DLQ, which is for runs (jobs) that fail — not for data rows. See
tratar-falha-dlq.md.
Schema drift: the mart does not break silently#
When the source schema changes, Ingestia compares the saved baseline with the current schema:
| Change | Severity | Effect |
|---|---|---|
| new column/table | info | additive — the mart does not break |
| removed column | atencao | the mart that reads it breaks |
| changed type | atencao | downstream cast/aggregation breaks |
| removed table | atencao | — |
The reported severity is the worst one found; the mart is marked as "breaks" when there is
atencao. This warns you before a dashboard shows a wrong number.
Safe profiling (a sample that does not leak)#
When cataloging a table, profiling calculates per column: declared × inferred type, % nulls, distinct values (estimate), min/max, length and up to 5 examples. Security is built in:
- PII column → examples become
•••; min/max/distinct/length suppressed. - Column hidden by governance → does not even appear (the name does not travel).
- Caps: 50 columns, example 60 chars, payload ≤ 32 KiB.
When to use / when not to use these guarantees#
- Rely on them to: measure the health of a source (rejection rate), avoid duplicates by key, and be warned about schema changes.
- Do not count on: a per-row quarantine queue with automatic resend (it does not
exist today), nor physical deletion propagated by pure watermark (see
conceitos-modos-de-carga.md).
Related#
- Bronze/Silver/Gold medallion pipeline
- Tutorial: create a quality validation
- Tutorial: build Bronze/Silver/Gold
- Tutorial: handle failures and the DLQ
Last reviewed: 2026-09-08.