Read first: Load modes · Reference: incremental and backfill.
What it is#
Backfill means bringing in the history older than the window the incremental load normally covers — for example, the last 24 months when the daily refresh only looks at the last few days.
What it is for#
- Populating a newly connected source with its entire history.
- Reprocessing a specific period after a rule fix.
- Extending a dashboard's time series further back.
When to use / when not to use#
- Use it when: you just connected the source, changed the business logic, or need an old period the incremental load does not reach.
- Do not use it when: the history is already loaded and you only want day-to-day data — then incremental by watermark is enough.
Plan and permissions#
- Core · access check · kill switches for ingestion runs and dashboard refresh.
Prerequisites#
- A connected and tested source.
- A clear idea of the period or volume you want to bring in.
The three ways to backfill#
| Method | How | When |
|---|---|---|
| One-off full load | run a full load once | covers the whole history at once; ideal for the 1st load |
| Wide window (by period) | increase recentDays (up to the 400 cap) | recalculates a larger window without a complete full load |
| Connector's first extraction | the pull's 1st run walks the pagination up to the cap | the 1st run itself is the backfill; later ones resume from the checkpoint/watermark |
Step by step#
- Decide the scope. The whole history? Do a full load. A specific period? Widen the window by period.
- Backfill by full load: open the source, make sure the tables are selected, and run the pipeline in full mode (see the full pipeline tutorial).
- Backfill by window: increase
recentDaysto cover the desired period (cap of 400 days) and run the refresh — it recalculates the widened window. - Backfill for a new connector: just run the first extraction; it walks the pagination up to the hard cap. Later runs continue from the checkpoint.
- After the backfill, go back to incremental for day-to-day loads (cheaper).
Example (synthetic)#
You connected a new pedidos table and want the last 2 years:
1) Run a FULL load once → brings the whole history up to the pagination cap.
2) Turn on INCREMENTAL → watermark column = atualizado_em.
3) Daily refresh → brings only the delta from then on.If you need to reprocess only January 2026 in a by-period aggregate:
Increase recentDays enough for the window to cover 2026-01 and run the refresh.How resuming works (checkpoint)#
In a long extraction that gets interrupted, the connector SDK uses a checkpoint:
- reads the checkpoint before extracting;
- writes it on each delivered batch (only accepted batches count);
- clears it on completion → the next scheduled run starts from scratch.
That way, resuming neither duplicates nor loses a row. Connectors such as Azure Blob skip blobs that were already processed (name ≤ last processed) to avoid reprocessing.
Expected result#
- The desired history shows up in Bronze and, after the pipeline, in the Gold marts.
- An extraction that hits the cap signals
hadMore(there is more at the source) — then you raise the cap deliberately or switch to incremental.
Limits and costs#
- Hard pagination cap: 10,000 pages / 5,000,000 rows.
- Window by period: cap of 400 days.
- Pull
maxRows: default 200,000 (up to 5,000,000). - A backfill reads a lot — it is the most expensive operation. Run it outside peak hours and go back to incremental afterward.
Security#
- Workspace scope on the entire load; secrets redacted in errors/logs.
- Checkpoint resume is scoped to the caller; no state leaks between customers.
Common errors#
| Symptom | Likely cause | What to do |
|---|---|---|
| "Incomplete" backfill | hit the pagination cap (hadMore) | Raise the cap deliberately or slice by period |
| Late data does not come in | arrived with a date outside the window | Increase recentDays or do a full load |
| Reprocessed twice | full load triggered on top of incremental | A full load rebuilds everything (idempotent); check the run history |
Troubleshooting#
- Check the run history:
bronzeRows/silverRowsand whetherhadMorewas signaled. - For connectors, look at observability (runs, freshness) to confirm that the source is "up to date".
Related#
Last reviewed: 2026-09-08.