Skip to content

Automatic translation — the Portuguese version is the reference. View in Portuguese

Updated on 2026-09-08

Tutorial — run a backfill (historical load)

Step-by-step guide to bringing in history older than the incremental window, through a one-off full load, a wide window, or a connector's first extraction.

On this page (15)

Read first: Load modes · Reference: incremental and backfill.


What it is#

Backfill means bringing in the history older than the window the incremental load normally covers — for example, the last 24 months when the daily refresh only looks at the last few days.

What it is for#

  • Populating a newly connected source with its entire history.
  • Reprocessing a specific period after a rule fix.
  • Extending a dashboard's time series further back.

When to use / when not to use#

  • Use it when: you just connected the source, changed the business logic, or need an old period the incremental load does not reach.
  • Do not use it when: the history is already loaded and you only want day-to-day data — then incremental by watermark is enough.

Plan and permissions#

  • Core · access check · kill switches for ingestion runs and dashboard refresh.

Prerequisites#

  1. A connected and tested source.
  2. A clear idea of the period or volume you want to bring in.

The three ways to backfill#

MethodHowWhen
One-off full loadrun a full load oncecovers the whole history at once; ideal for the 1st load
Wide window (by period)increase recentDays (up to the 400 cap)recalculates a larger window without a complete full load
Connector's first extractionthe pull's 1st run walks the pagination up to the capthe 1st run itself is the backfill; later ones resume from the checkpoint/watermark

Step by step#

  1. Decide the scope. The whole history? Do a full load. A specific period? Widen the window by period.
  2. Backfill by full load: open the source, make sure the tables are selected, and run the pipeline in full mode (see the full pipeline tutorial).
  3. Backfill by window: increase recentDays to cover the desired period (cap of 400 days) and run the refresh — it recalculates the widened window.
  4. Backfill for a new connector: just run the first extraction; it walks the pagination up to the hard cap. Later runs continue from the checkpoint.
  5. After the backfill, go back to incremental for day-to-day loads (cheaper).

Example (synthetic)#

You connected a new pedidos table and want the last 2 years:

code
1) Run a FULL load once → brings the whole history up to the pagination cap.
2) Turn on INCREMENTAL → watermark column = atualizado_em.
3) Daily refresh → brings only the delta from then on.

If you need to reprocess only January 2026 in a by-period aggregate:

code
Increase recentDays enough for the window to cover 2026-01 and run the refresh.

How resuming works (checkpoint)#

In a long extraction that gets interrupted, the connector SDK uses a checkpoint:

  • reads the checkpoint before extracting;
  • writes it on each delivered batch (only accepted batches count);
  • clears it on completion → the next scheduled run starts from scratch.

That way, resuming neither duplicates nor loses a row. Connectors such as Azure Blob skip blobs that were already processed (name ≤ last processed) to avoid reprocessing.

Expected result#

  • The desired history shows up in Bronze and, after the pipeline, in the Gold marts.
  • An extraction that hits the cap signals hadMore (there is more at the source) — then you raise the cap deliberately or switch to incremental.

Limits and costs#

  • Hard pagination cap: 10,000 pages / 5,000,000 rows.
  • Window by period: cap of 400 days.
  • Pull maxRows: default 200,000 (up to 5,000,000).
  • A backfill reads a lot — it is the most expensive operation. Run it outside peak hours and go back to incremental afterward.

Security#

  • Workspace scope on the entire load; secrets redacted in errors/logs.
  • Checkpoint resume is scoped to the caller; no state leaks between customers.

Common errors#

SymptomLikely causeWhat to do
"Incomplete" backfillhit the pagination cap (hadMore)Raise the cap deliberately or slice by period
Late data does not come inarrived with a date outside the windowIncrease recentDays or do a full load
Reprocessed twicefull load triggered on top of incrementalA full load rebuilds everything (idempotent); check the run history

Troubleshooting#

  • Check the run history: bronzeRows/silverRows and whether hadMore was signaled.
  • For connectors, look at observability (runs, freshness) to confirm that the source is "up to date".

Related#


Last reviewed: 2026-09-08.

Related links