Skip to content

Automatic translation — the Portuguese version is the reference. View in Portuguese

Scheduling, retry, DLQ and reconciliation

On this page (7)

Read first: README.md. Layers in pipeline-medalhao.md.


Scheduling#

  • The schedule time is chosen and shown in Brasília time, on the pipeline screen (ETL/ELT → Pipelines → the pipeline → Schedule). See Schedule and time zone.
  • The system checks schedules every 15 minutes: a pipeline set for 07:00 starts between 07:00 and 07:15.
  • The next run is calculated as soon as you save or edit the schedule — that is what appears on the pipeline screen.
  • Runs that get stuck are terminated automatically and count as failures (they fall under the retry rules below).

Retry / backoff#

Pure central policy (no I/O); the ones that write to the database are the dispatchers, with a conditional updateMany (atomic claim). No new column (frozen schema): the retry state lives in markers in the existing error field:

code
[tentativa N] [apos <ISO>] <mensagem original>
  • [tentativa N] — the reaper's current marker (format preserved byte for byte).
  • [apos <ISO>] — "do not run before" (implements backoff without a schedule column).

Attempt cap per type (semantics differ, inherited from the models):

TypeMAX_ATTEMPTSCounts
dashboard2requeues via [tentativa N] (2 requeues = up to 3 runs)
report3runs (attempt column, incremented on claim)
pipeline2consecutive failed runs of the same source

Backoff: exponential base × 2^(n-1) with deterministic jitter (FNV-1a hash of the seed, e.g. runId). Base = 5 min (1 cron tick), cap = 1 h, jitter up to 25%. Deterministic on purpose: the test proves the exact value and production does not depend on Math.random.

Failure classification:

  • poison (config/validation — retrying won't fix it) → straight to DLQ, without burning attempts. Conservative patterns: not found, syntax error, unrecognized name, invalid_grant, access/permission denied, unauthorized, forbidden, sem saldo de créditos, sem consultas configuradas.
  • retryable (transient/unknown) → retries up to the cap (unknown errors stay retryable; the cap protects).

The post-failure decision returns {acao:"retry", tentativa, delayMs, executarApos} or {acao:"dlq", motivo}.

DLQ — Dead Letter Queue#

DLQ = status "dlq" (new status value). A run in DLQ leaves all executable filters (dispatchers read queued, the reaper reads running) and only comes back by a human's hand: reprocessing from the DLQ is a manual intervention by an operator, with auditing — never automatic.

A run goes to DLQ when: a poison error occurs, or the attempt cap is reached.

In practice — what you see

Verified end to end in the full test of 10/06/2026, with a scheduled pipeline whose source was down:

  1. The scheduled run fails → the next attempt is brought forward to a few minutes from now (the wait comes from the pipeline's Operation › Retries; the default is 1 retry with a 5 min wait).
  2. The attempt also fails → the run goes to DLQ and the audit log records job.dlq with the reason ("attempt cap reached"). The schedule carries on as normal: the next time slot on the following day stays scheduled.
  3. To reprocess what landed in the DLQ, open a ticket (or ask Aiô): the reprocessing is done by the team, with auditing. On the screen, the equivalent path is Runs › Recover… › Run the pipeline again.

A partial run counts as a failure, but does not run earlier

If only one task fails (for example, a Silver with broken SQL) and the others finish, the run is partial. On a scheduled run, a partial run counts toward the retry limit like a failure: consecutive partial runs send the run to the DLQ, and a configuration error goes to the DLQ right away. The difference is that a partial run does not bring the next run forward: some tables already loaded, so the new attempt is the next scheduled time itself. The failure notice is sent; to fix it sooner, adjust the task and use Recover… › New task attempt on that task only.

Layer reconciliation (read-only)#

A kill midway leaves a gap between layers (bronze loaded without silver, gold older than silver, promised bronze that never arrived). The product compares the three trails (pipeline run, Bronze tables and transformations) and produces an actionable report — the fix is up to the operator (zero automatic fixes in this wave).

Rules (always conditioned on there being a real expectation):

  • Bronze — source with an ok run + selected table: never processed → gap; load older than the start of the last ok run → stale load.
  • Silver — there is processed bronze: transform never run → bronze→silver gap; older than the most recent bronze → outdated.
  • Gold — there is processed silver: transform never run → silver→gold gap; older than the most recent silver → outdated.

State collection is entirely scoped by workspaceId (invariant: one customer's stamp never reads another's).

Per-connector observability#

Aggregates extraction events (one row per operation) into runs, error rate, p50/p95, last run and freshness:

ThresholdValueEffect
error rate≥ 20%degraded
p95> 30 sdegraded
freshness> 26 hdegraded
freshness> 72 hstopped

No events at all → stopped.


Fact sheet (9-field template)#

  1. State — available. Scheduling, backoff, DLQ and reconciliation are covered by automated tests.
  2. What it does — schedules loads, retries transient failures with backoff, isolates permanent failures in DLQ and points out gaps between layers.
  3. How it works — Postgres + 5-min crons; precomputed nextRunAt; markers in the error field; DLQ = terminal status; read-only reconciliation per workspace.
  4. Plan · Permission — core · access check · kill switches for ingestion runs / report runs / dashboard refresh.
  5. Cost — retry/backoff reduces the cost of blind re-runs; jitter avoids hammering. No platform cost in the calculation.
  6. Limits — attempts: dashboard 2 / report 3 / pipeline 2; backoff base 5 min, cap 1 h, jitter 25%.
  7. Security — atomic claim (conditional updateMany) prevents races; DLQ requires audited human intervention; reconciliation scoped by workspace.
  8. How to validate — in the console, schedule a load and follow the nextRunAt and the run history; force a transient failure and confirm the rescheduling with backoff; a configuration failure goes straight to the DLQ, which it only leaves through an audited manual action.
  9. In short — idempotency, retry with backoff, DLQ and layer reconciliation.

Related links