Read first: Scheduling, retry, DLQ, and reconciliation.
What it is#
Not every run ends well. Ingestia handles failures at three levels: it retries whatever is transient (with backoff), isolates whatever is permanent in a DLQ (Dead Letter Queue), and points out gaps between layers after an interruption. This tutorial shows how to read and act on each one.
The DLQ is for runs (jobs) that fail — not for data rows. Rows that fail quality checks are something else (see Quality).
What it is for#
Preventing a temporary failure from bringing the flow down (retry solves it on its own) and making sure a permanent failure does not stay in a loop consuming resources (it goes to the DLQ and waits for a person).
When to use / when not to use#
- Use this guide when a refresh/report/pipeline fails or shows up in the DLQ.
- Do not blindly reprocess a permanent failure without fixing the cause — it will go back to the DLQ.
Plan and permissions#
- Core · access check · kill switches for ingestion runs, report runs, and dashboard refresh.
- Reprocessing from the DLQ is a manual, audited operator action: there is no button on the screen — ask through Aiô or open a ticket. To run it again yourself, use Runs › Recover… › Run the pipeline again.
How Ingestia classifies the failure#
| Class | Examples | What happens |
|---|---|---|
| retryable (transient/unknown) | timeout, 5xx, network error | retries with backoff up to the cap |
| poison (config/validation — retrying will not fix it) | not found, syntax error, access denied, invalid_grant, sem saldo de créditos | goes straight to the DLQ, without burning attempts |
Retry with backoff (what you see)#
- Attempt cap by type: dashboard 2 · report 3 · pipeline 2.
- Exponential backoff
base × 2^(n-1)with deterministic jitter; base 5 min (1 cron tick), cap 1 h, jitter up to 25%. - The retry state appears as markers in the error field, e.g.:
[tentativa 2] [apos 2026-09-08T13:40:00Z] connection reset by peer[tentativa N] = how many times it has already retried; [apos <ISO>] = "do not run before".
Step by step#
Case 1 — transient failure (retry solves it)
- You see
[tentativa N]and a[apos...]time on the run. - Wait: the dispatcher reruns it after the backoff, on its own.
- If things go back to normal, there is nothing to do. If it exceeds the cap, it becomes a DLQ item (case 2).
Case 2 — permanent failure (DLQ)
- The run shows status
dlqand leaves all runnable filters. - Read the reason (the redacted message says what happened: permission, syntax, credits, etc.).
- Fix the cause at the source: credential, permission, SQL, balance, configuration.
- Reprocess: ask support (through Aiô or a ticket) to reprocess from the DLQ — an operator action, audited, never automatic — or run it again from the screen in Runs › Recover… › Run the pipeline again.
Case 3 — interruption midway (reconciliation)
A kill midway can leave a gap between layers (Bronze without Silver, Gold older than Silver). Ingestia compares the three trails (run, Bronze, transformations) and produces an actionable report per workspace. The fix is up to the operator (no automatic fix today).
Example (synthetic)#
A dashboard refresh failed because of an expired connector token:
Error: [tentativa 1] invalid_grant: token expired
Class: poison (invalid_grant) → straight to the DLQ
Action: renew the source credential → reprocess from the DLQA pipeline failed because of a network timeout:
Error: [tentativa 1] [apos 2026-09-08T13:35:00Z] ETIMEDOUT
Class: retryable → retries in ~5 min (backoff)
Action: wait; if it exceeds the cap (2), it goes to the DLQExpected result#
- Transient failures resolve on their own within the attempt cap.
- Permanent failures stay stopped in the DLQ (they do not burn resources) until they are fixed + reprocessed with an audit trail.
- After an interruption, the reconciliation report points out exactly where the gap is.
Limits and costs#
- Attempts: dashboard 2 / report 3 / pipeline 2. Backoff base 5 min, cap 1 h.
- Retry with backoff reduces the cost of blind reruns; the jitter avoids "hammering" the source.
Security#
- An atomic claim (conditional
updateMany) prevents two dispatchers from taking the same run. - The DLQ requires audited human intervention to reprocess.
- Reconciliation is read-only and scoped by
workspaceId— one customer's stamp never reads another's. Error messages have their secrets redacted.
Common errors#
| Symptom | Likely cause | What to do |
|---|---|---|
| Run "stuck" in running | the process died midway | The "reaper" recovers it; reconciliation points out the gap |
| Keeps going back to the DLQ | permanent cause not fixed | Fix the source (permission/SQL/credits) before reprocessing |
| Never retries | error classified as poison | This is expected: poison goes straight to the DLQ |
Troubleshooting#
- Read the
[tentativa N]/[apos...]markers to understand where in the retry cycle the run is. - Use the reconciliation report after any interruption.
- Check the connector's observability (error rate ≥ 20% → degraded; freshness > 26 h → degraded, > 72 h → stopped).
Related#
- Scheduling, retry, DLQ, and reconciliation (reference)
- Quality, deduplication, and rejected rows
- Create a quality validation
Last reviewed: 2026-09-08.