Skip to content

Automatic translation — the Portuguese version is the reference. View in Portuguese

Updated on 2026-09-08

Tutorial — handle failures and the DLQ (Dead Letter Queue)

Step-by-step guide to understanding retry with backoff, permanent failures in the DLQ, and how to reprocess in an audited way.

On this page (17)

Read first: Scheduling, retry, DLQ, and reconciliation.


What it is#

Not every run ends well. Ingestia handles failures at three levels: it retries whatever is transient (with backoff), isolates whatever is permanent in a DLQ (Dead Letter Queue), and points out gaps between layers after an interruption. This tutorial shows how to read and act on each one.

The DLQ is for runs (jobs) that fail — not for data rows. Rows that fail quality checks are something else (see Quality).

What it is for#

Preventing a temporary failure from bringing the flow down (retry solves it on its own) and making sure a permanent failure does not stay in a loop consuming resources (it goes to the DLQ and waits for a person).

When to use / when not to use#

  • Use this guide when a refresh/report/pipeline fails or shows up in the DLQ.
  • Do not blindly reprocess a permanent failure without fixing the cause — it will go back to the DLQ.

Plan and permissions#

  • Core · access check · kill switches for ingestion runs, report runs, and dashboard refresh.
  • Reprocessing from the DLQ is a manual, audited operator action: there is no button on the screen — ask through Aiô or open a ticket. To run it again yourself, use Runs › Recover… › Run the pipeline again.

How Ingestia classifies the failure#

ClassExamplesWhat happens
retryable (transient/unknown)timeout, 5xx, network errorretries with backoff up to the cap
poison (config/validation — retrying will not fix it)not found, syntax error, access denied, invalid_grant, sem saldo de créditosgoes straight to the DLQ, without burning attempts

Retry with backoff (what you see)#

  • Attempt cap by type: dashboard 2 · report 3 · pipeline 2.
  • Exponential backoff base × 2^(n-1) with deterministic jitter; base 5 min (1 cron tick), cap 1 h, jitter up to 25%.
  • The retry state appears as markers in the error field, e.g.:
code
[tentativa 2] [apos 2026-09-08T13:40:00Z] connection reset by peer

[tentativa N] = how many times it has already retried; [apos <ISO>] = "do not run before".

Step by step#

Case 1 — transient failure (retry solves it)

  1. You see [tentativa N] and a [apos...] time on the run.
  2. Wait: the dispatcher reruns it after the backoff, on its own.
  3. If things go back to normal, there is nothing to do. If it exceeds the cap, it becomes a DLQ item (case 2).

Case 2 — permanent failure (DLQ)

  1. The run shows status dlq and leaves all runnable filters.
  2. Read the reason (the redacted message says what happened: permission, syntax, credits, etc.).
  3. Fix the cause at the source: credential, permission, SQL, balance, configuration.
  4. Reprocess: ask support (through Aiô or a ticket) to reprocess from the DLQ — an operator action, audited, never automatic — or run it again from the screen in Runs › Recover… › Run the pipeline again.

Case 3 — interruption midway (reconciliation)

A kill midway can leave a gap between layers (Bronze without Silver, Gold older than Silver). Ingestia compares the three trails (run, Bronze, transformations) and produces an actionable report per workspace. The fix is up to the operator (no automatic fix today).

Example (synthetic)#

A dashboard refresh failed because of an expired connector token:

code
Error: [tentativa 1] invalid_grant: token expired
Class: poison (invalid_grant) → straight to the DLQ
Action: renew the source credential → reprocess from the DLQ

A pipeline failed because of a network timeout:

code
Error: [tentativa 1] [apos 2026-09-08T13:35:00Z] ETIMEDOUT
Class: retryable → retries in ~5 min (backoff)
Action: wait; if it exceeds the cap (2), it goes to the DLQ

Expected result#

  • Transient failures resolve on their own within the attempt cap.
  • Permanent failures stay stopped in the DLQ (they do not burn resources) until they are fixed + reprocessed with an audit trail.
  • After an interruption, the reconciliation report points out exactly where the gap is.

Limits and costs#

  • Attempts: dashboard 2 / report 3 / pipeline 2. Backoff base 5 min, cap 1 h.
  • Retry with backoff reduces the cost of blind reruns; the jitter avoids "hammering" the source.

Security#

  • An atomic claim (conditional updateMany) prevents two dispatchers from taking the same run.
  • The DLQ requires audited human intervention to reprocess.
  • Reconciliation is read-only and scoped by workspaceId — one customer's stamp never reads another's. Error messages have their secrets redacted.

Common errors#

SymptomLikely causeWhat to do
Run "stuck" in runningthe process died midwayThe "reaper" recovers it; reconciliation points out the gap
Keeps going back to the DLQpermanent cause not fixedFix the source (permission/SQL/credits) before reprocessing
Never retrieserror classified as poisonThis is expected: poison goes straight to the DLQ

Troubleshooting#

  • Read the [tentativa N]/[apos...] markers to understand where in the retry cycle the run is.
  • Use the reconciliation report after any interruption.
  • Check the connector's observability (error rate ≥ 20% → degraded; freshness > 26 h → degraded, > 72 h → stopped).

Related#


Last reviewed: 2026-09-08.

Related links