Orchestration is what happens in the ETL/ELT group of the menu (Connections · Pipelines · Runs). This is where a pipeline becomes a published version, gets (or doesn't get) a schedule, is run, and, when it fails, is recovered.
The separation matters, and the product takes it seriously:
| Object | What it is | Where it lives |
|---|---|---|
| Connection | Authorized access to a source: address, parameters, and the encrypted credential. | ETL/ELT → Connections |
| Pipeline | The definition of what to read, transform, and write. It points to a connection; it never copies the credential. | ETL/ELT → Pipelines |
| Version | The immutable snapshot of the definition at the moment of publishing. | The pipeline's publishing panel |
| Schedule | The rule for when to run. You choose the time in Brasília time; it stays off until you turn it on. | Pipeline schedule card |
| Run | A concrete round of that version, with tasks, attempts, and logs. | ETL/ELT → Runs |
Publishing is not running
The first publication of a pipeline creates version 1 and leaves the schedule off. Nothing is read, written, or charged at that moment. Turn on schedule and Run now are two separate actions, each with its own click.
Guides for this product#
Publish and version a pipeline
The first publication, what changes when you update a pipeline that is already active, and what the version freezes.
Schedule, time zone, and pause
Turn on the schedule, run now, pause — and the Brasília time in which everything is shown.
Track a run
From the run to the task, from the task to the attempt, from the attempt to the right log.
Recover from a failure
The four recovery scopes and the preview that matches the real effect.
Day-to-day operations (articles in Data)#
These topics belong to orchestration, but the articles have lived in Data since the first version of the documentation. The addresses still work — nothing was moved, nothing broke:
- Scheduling, retry, and DLQ — windows, retry policy, and dead-letter queue.
- Incremental ingestion and backfill — watermark, checkpoint, and reprocessing a period.
- Medallion pipeline — Bronze, Silver, and Gold.
- Load modes — full, incremental, append, merge, replace.
- Pipeline tutorials — full, incremental with watermark, backfill, and failure handling.
Before you start#
- You must be the workspace owner to see Pipelines and Runs. People who only consume BI don't see these screens.
- Running a pipeline is an expensive action: with the subscription overdue, suspended, or canceled, the run is blocked with an explicit message and a link to Plans. See Suspension.
- The connection test goes out from the application server; the pipeline executor runs on another network, with another outbound IP. Allow both on your network.
If something goes wrong#
| Symptom | Next action |
|---|---|
| I published and nothing happened | That is the expected behavior: use Turn on schedule or Run now. |
| The schedule didn't fire | Check that it is on (the Schedule on badge) and the Brasília time shown on the pipeline. |
| The run failed | Open the task, then the attempt, then the log — in that order. |
| I reran and it duplicated | Check the scope chosen in the recovery and the pipeline's load mode. |
| Expired credential | Fix it in the connection, once. Every pipeline that points to it starts using the new value. |
Last review: 2026-09-16. Examples use the fictitious workspace Comércio
Aurora Ltda (aurora). No customer data appears in this documentation.