This page is an interface inventory: the text here is the text on screen, literally. To learn the task with screenshots, go to First ingestion.
Where it is: menu ETL/ELT → Pipelines.
A pipeline brings data from a system or file into the datalake, where it becomes dashboards.
Create a pipeline — guided mode (/sources/new)#
New pipeline opens guided mode: four questions.
| Question | What you choose |
|---|---|
| 1 · Where does the data come from? | One of Your connections, Upload a file or spreadsheet (CSV, JSON, Parquet), or Connect a database (registers the connection and comes back) |
| 2 · What to bring in? | Database: check the tables (there's a search). File: upload the files and choose Replace with the new one or Combine with the previous ones |
| 3 · When to update? | Only when I say so, Every day, at a set time (Brasília time), or, for files, Whenever I upload a new file |
| 4 · Review and create | Check the summary sentence and the Pipeline name |
In step 4: Create and run now (creates it, turns on the chosen update, and runs the first load right away) or Just create. At the end, the Pipeline created! screen offers Follow the run, Open the pipeline, and Back to the list.
Guided mode brings in the whole table on every run. Loading only what changed,
transformations (Silver), partitioning, sensitive columns, alerts, and tools
(Google Sheets, Meta Ads, CRM) are in full mode: the Use full
mode link at the top (/sources/new?modo=avancado). New report, in the list's
bar, also opens full mode.
Register the connection before building the pipeline#
The pipeline points to a saved connection; it doesn't store a password. In guided mode, Connect a database opens the four-step wizard (System, Address and password, Test, Done) and, when you save, you come back with the connection already selected. Details of each field in Connections screens.
Create a pipeline in full mode#
- In ETL/ELT → Pipelines, click New pipeline and then Use full mode (or New report).
- Under What do you want to create?, choose Start extraction or Create Gold report.
- Go through the six stages — Connections, Data, Load, Transformations, Operation, Review — moving ahead with Continue to Data, Continue to Load, and so on.
- In the Review stage, click Publish pipeline. It stays disabled while the review lists any problem — the Fix in "{etapa}" link takes you to it.
- Then Run now runs the first load; Turn on schedule turns on automation.
Publishing does not load anything: it creates the version and leaves the schedule off.
Go through the six building stages#
Connections → Data → Load → Transformations → Operation → Review
| Stage | What you decide | Button to move ahead |
|---|---|---|
| 1 · Connections | the saved connection, in Your connections | Continue to Data |
| 2 · Data | the tables, files, or endpoints | Continue to Load |
| 3 · Load | Full or Append, and the Control key column | Continue to Transformations |
| 4 · Transformations | the Silver tables and the dependencies | Continue to Operation |
| 5 · Operation | schedule, retries, limits, and alerts | Continue to Review |
| 6 · Review | check everything | Publish pipeline |
At any stage, Save draft keeps what you've built — and the button itself explains the limit: "Keeps your setup. Doesn't publish, run or turn on the schedule."
Choose between Full and Append (stage 3)#
- Full — "Reloads the whole table on every run (replaces everything)."
- Append — "Adds only new data on every run (incremental)." It requires a Control key column: a reliable date or sequence.
The Write at the destination — actual effect card shows, before publishing, the Write, Deduplication, History outside the window, Deletes at the source, and Schema change lines. Start with Full and validate incremental later.
Configure the schedule while building (stage 5)#
In Pipeline schedule, the Mode field has two options: Scheduled ("Runs at a fixed time (daily or D-1), with a reprocessing window.") and On file arrival ("Runs as soon as the file arrives in the landing area.").
Frequency is Daily or D-1 (previous day), with Time (Brasília) and Days to look back (the latter with the not applied yet badge: it's recorded in the run, but today it doesn't change what gets loaded). The Next five runs card shows the upcoming triggers, in Brasília time, before turning anything on.
Run limits — they really apply
Since 10/03/2026, the four Run limits fields change what happens during the run:
| Field | What happens |
|---|---|
| Retry on failure | How many times an automatic (scheduled) run that failed is repeated. 0 = no repeat. |
| Wait between retries (min) | How long to wait before trying again. |
| Timeout (min) | Past this, the run is ended as failed (checked every 10 min). |
| Concurrent runs | How many runs of this pipeline can run at the same time. 1 is recommended for a load that rewrites the table. |
A manual run doesn't repeat on its own
Retries apply to the schedule. If a run you started fails, use "Try again" in the Runs list. And when the concurrent runs limit is reached, Run now is refused with the reason.
The Notifications and owners card closes the stage: who gets notified and who is the owner. Publishing doesn't send a message to anyone — only an actual run triggers email.
Publish the pipeline (stage 6)#
The button is Publish pipeline (while running: "Publishing..."). Next to it, before you click, the screen tells you what will happen: "Publishing creates the pipeline version. The schedule will stay off and no load will start."
Publishing, scheduling, and running are three different clicks
Once published, Pipeline published — version {N} appears and, with it, three separate actions:
- Run now — "Triggers ONE run. Doesn't turn on the schedule."
- Turn on schedule — "Turns on scheduling. Doesn't run anything now."
- Open operation
Run the first load#
Use Run now, in the panel that appears right after publishing or in the pipeline's operation bar. Turn on schedule is another decision, for later — and turning on the schedule does not recover history.
To track what ran, go to ETL/ELT → Runs (Runs screens).
Publish a new version of an existing pipeline#
On the pipeline screen, the operation bar shows the Version {N} published badge (or Version not recorded), the schedule state, and the Unpublished changes badge when there's something to publish. The actions:
- Review and publish — "Shows source, target, write mode, schedule, limits, and what changes". Inside the review, the final button is Publish version {N+1} — or Nothing to publish, when nothing changed.
- Turn on schedule — becomes Pause schedule after you turn it on
- Run now
Operate the five stages of the pipeline screen#
| # | Stage | What it shows | Action |
|---|---|---|---|
| 1 | Connection | where the data comes from | Open connection |
| 2 | Scheduling | when the pipeline runs | Edit |
| 3 | Bronze layer | the raw data, identical to the source | Ingest live |
| 4 | Silver layer / Gold layer | the clean data / ready for analysis | New Silver · New Gold · Notebook |
| 5 | Alerts and details | who gets notified when it runs | Edit |
Every action button becomes Close while the stage is open.
Ingest live is not Run now
Run now triggers the run through orchestration, with history and retries. Ingest live runs the ingestion showing each step on screen: it's for you to watch it happen, not for day-to-day operation.
Configure table, watermark, and alerts#
In stage 3, per table: Ingestion type (Full (reloads the whole table) / Append (adds new data only)), Key column (watermark), Partition column (optional), Clustering (up to 4 columns — optional), and Technical columns (lineage).
In stage 5: Source name *, Owner, Description, and the Email alerts block, with separate Success emails and FAILURE emails — saved with Save details.
Find a pipeline in the list (/sources)#
Title Pipelines, subtitle "Each pipeline connects a source (database, spreadsheet, or platform) and transforms it into Bronze → Silver → Gold in your datalake."
- Search: "Search pipeline by name…" · Filter: All types
- Counter: "{N} pipelines · {N} extraction · {N} Gold reports"
- Columns: Pipeline · Status · Schedule · Last sync · Monthly volume · target · Actions
Each row has the type badge — Extraction or Gold report — and the status: Connected, Paused, Error, Syncing, or Last run had failures. With no load at all, the date says Never synced.
Empty list: No sources connected + the Connect source button. Filter with no results: "No pipeline found for this filter."
Change the time of an existing pipeline#
On the pipeline screen, the Pipeline schedule card has the Edit schedule button. The Mode has three options:
- Only when I run it — no schedule; use Run now
- Every day, at a set time — with Frequency (Daily or D-1 (previous day)), Time (Brasília), and Days to look back
- When a new file arrives — only for file sources
Save with Save schedule. The card warns about the switch state: "Schedule ON — saving here changes the time, it doesn't turn anything off." or "Schedule OFF — nothing runs automatically until you turn it on…". Turning it on and off is always at the top: Turn on schedule / Pause schedule.
Micro-batch: every 5 or 15 minutes
With Every day, at a set time, the Cadence field picks the pace: Batch (once a day) or Micro-batch (minutes) — the latter with an Interval of Every 5 minutes or Every 15 minutes, on the Business and Scale plans. Each run brings only what changed. A micro-batch pipeline that stops running at the promised pace shows as Behind. See Micro-batch.
Notebook step in Silver or Gold
The Notebook button, next to New Silver / New Gold, creates a step that runs a published notebook: Published notebook, Version, Target table, Load, Where it runs and Parameters, saved with Save step. See Notebook step.
What the schedule says about itself#
| Badge | When it appears |
|---|---|
| Schedule on | there's a time and the switch is on |
| Schedule paused | there's a time and the switch is off |
| On event | the frequency is On file arrival |
| Manual | there's no frequency or time — the switch on, by itself, doesn't create a schedule |
Back to the app#
- Pipelines: ETL/ELT → Pipelines
- Build one: New pipeline
Next task#
- First ingestion — the tutorial
- Runs screens — track what ran
- Publish and version