Every pipeline has a cadence: how often it fetches data from the source.
- Batch (once a day) — the default. The pipeline runs at a time of day (or for the previous day, D-1).
- Micro-batch (minutes) — the pipeline runs every 5 or every 15 minutes and, on each round, brings only what changed since the previous one. It is the product's near real time option: the dashboard stays, at most, a few minutes behind the source.
Which plan
Micro-batch is part of the Business and Scale plans. On the other plans the cadence stays at Batch; if you choose micro-batch, the screen refuses with the notice "Micro-batch is not included in your plan." See Plans.
Turn on micro-batch#
1. Open the pipeline
In ETL/ELT → Pipelines, open the pipeline and go to the Pipeline schedule card. Click Edit schedule.
2. Choose the cadence
In Mode, keep Every day, at a set time. In Cadence, choose Micro-batch (minutes).
3. Choose the interval
In Interval, choose Every 5 minutes or Every 15 minutes.
4. Save and check the schedule
Click Save schedule. Micro-batch only runs with the schedule on (Pause/Activate schedule, at the top of the pipeline screen).
To go back to the daily rhythm, switch Cadence to Batch (once a day) and save.
What micro-batch reads on each round#
Micro-batch does not reload the whole table every 5 minutes. Each round
brings only what changed since the previous one, using the table's key
column (a date/time column that only grows, such as atualizado_em) — the
same incremental load described in Load modes.
| The table… | With micro-batch on |
|---|---|
| is incremental (appends) and has a date/time key column | runs at every interval, bringing only new or changed rows |
| has no key column, or is reloaded in full | stays on the batch rhythm: loads once a day. The minute round skips that table and records the reason |
So turning on micro-batch in a pipeline with tables that have no key column does not multiply the cost: only tables that can be sliced run every few minutes.
To get the most out of it
On the Bronze table, use the incremental load and choose a source update date/time as the key column. See Incremental ingestion and backfill.
The delay warning#
A pipeline that promises minutes and stops running is flagged right away, not just the next day. When the last run falls more than three intervals behind, the pipeline shows as Behind — in the pipeline list, in the pipeline detail and on the Operations screen — with the number spelled out:
"Promised every 5 min, but the last run was 40 min ago — 8 periods without running. Check the schedule and the runs."
- A paused pipeline is not flagged: pausing switches the promise off.
- A pipeline that never ran is not behind — it shows as "never run".
- A daily batch pipeline does not have this warning.
If the warning appears, open ETL/ELT → Runs and check the last round. The most common causes are a depleted credit balance, a run that takes longer than the interval (the next one waits for the previous to finish) and an expired source credential. See Scheduling, retry and DLQ.
Cost#
Each round is a pipeline run and consumes credits like any other — proportional to what it reads. At 5 minutes that is up to 288 rounds a day; at 15, up to 96. Start with 15 minutes and only go down to 5 if the business needs it. Track it in Settings → Costs (Usage and quotas).
Micro-batch is not real time#
Micro-batch fetches data at short intervals. It does not read the database's transaction log (CDC) and, therefore:
- it does not see a row deleted at the source (the incremental load only sees what was inserted or changed);
- the dashboard stays a few minutes behind the source, never seconds.
Continuous change capture (CDC/real time) is not available yet. For near real time, micro-batch is the way. See How data flows.