Skip to content

Automatic translation — the Portuguese version is the reference. View in Portuguese

Updated on 2026-09-08

Quality, deduplication, rejected rows and quarantine

How Ingestia cleans data in Silver — quality rules, deduplication, rejected-row counts and the role of Bronze as retention of the raw data.

On this page (9)

Read first: conceitos-fluxo-de-dados.md. Silver layer reference in pipeline-medalhao.md.


The principle: raw data is never lost#

Ingestia's quality checks are applied on the Bronze → Silver step. The core guarantee is that Bronze is immutable and verbatim: each source row enters as text, with no conversion, with audit stamps. Therefore, no row disappears — even a row rejected during cleaning still exists in Bronze, available for inspection and reprocessing.

code
Bronze (everything, raw) → quality rules + dedupe → Silver (only valid, clean)
                                    │
                                    └──▶ rejected = Bronze − Silver (counted, visible)

The quality rules (reference example)#

The reference implementation (synthetic vendas table) applies the rules row by row. A failure = the row does not enter Silver and is counted in rejectedRows:

RuleBehavior
Status canceladodiscarded
Quantityinteger > 0, otherwise discarded
Priceaccepts BR (1.299,90) and US (199.90) formats; > 0, otherwise discarded
Datevalid YYYY-MM-DD, otherwise discarded; derives mes = YYYY-MM
State (UF)normalized to 2 uppercase letters, otherwise discarded
pedido_idrequired
Empty categorybecomes "Sem categoria" (not discarded)
Deduplicationby pedido_id, keeps the 1st occurrence
receitaderived = price × quantity

Note the difference between discarding (the row failed a hard rule) and fixing (empty category becomes "Sem categoria" — the row is kept). The product fixes when it is safe and discards when it is not.

Rejected rows: the difference is always visible#

The number of rejected rows is always explicit:

code
rejectedRows = bronzeRows − silverRows

Each run records this trio (Bronze / Silver / rejected) in the run history. You see, on every run, how many rows came in raw, how many survived cleaning and how many were left out. Nothing is discarded silently.

Deduplication#

Deduplication happens in Silver, by business key (in the example, pedido_id), keeping the first occurrence. Subsequent duplicates of the same key are discarded (and count toward the rejected total). This makes Silver idempotent by key, even if the source sends the same row more than once.

"Quarantine": what the product actually does (honesty)#

There is no separate quarantine table today for rows rejected in the Silver layer. What exists — and plays the same role — is the combination of:

  1. Verbatim Bronze keeps all raw rows (the "retention" of what arrived);
  2. the rejected counter (bronzeRows − silverRows) makes the loss measurable;
  3. automatic load sanitization preserves problematic file rows instead of inventing data (an unparseable row is kept as is).

In other words: you do not lose the rejected data (it is in Bronze) and you know how much was rejected (counter), but Ingestia does not promise today a dedicated quarantine queue with a per-row fix/resend flow. Do not confuse this with the DLQ, which is for runs (jobs) that fail — not for data rows. See tratar-falha-dlq.md.

Schema drift: the mart does not break silently#

When the source schema changes, Ingestia compares the saved baseline with the current schema:

ChangeSeverityEffect
new column/tableinfoadditive — the mart does not break
removed columnatencaothe mart that reads it breaks
changed typeatencaodownstream cast/aggregation breaks
removed tableatencao—

The reported severity is the worst one found; the mart is marked as "breaks" when there is atencao. This warns you before a dashboard shows a wrong number.

Safe profiling (a sample that does not leak)#

When cataloging a table, profiling calculates per column: declared × inferred type, % nulls, distinct values (estimate), min/max, length and up to 5 examples. Security is built in:

  • PII column → examples become •••; min/max/distinct/length suppressed.
  • Column hidden by governance → does not even appear (the name does not travel).
  • Caps: 50 columns, example 60 chars, payload ≤ 32 KiB.

When to use / when not to use these guarantees#

  • Rely on them to: measure the health of a source (rejection rate), avoid duplicates by key, and be warned about schema changes.
  • Do not count on: a per-row quarantine queue with automatic resend (it does not exist today), nor physical deletion propagated by pure watermark (see conceitos-modos-de-carga.md).

Related#


Last reviewed: 2026-09-08.

Related links