Besides guided and free SQL steps, a Silver or Gold table can be produced by a notebook. The pipeline runs the notebook's published version as a Spark job, in order with the other steps, every time the pipeline runs — on schedule, through Run now or when reprocessing a period.
First: create and publish the notebook. See Spark notebooks and clusters.
Create the step#
1. Open the layer
On the pipeline detail, go to the Silver (or Gold) layer. Next to New Silver (or New Gold), click Notebook.
2. Choose the notebook
In Published notebook, choose the notebook and, in Version, the version the pipeline will run (the most recent comes selected).
3. Say where to write
In Target table, name the table the step produces. In Load, choose Replace the table or Append rows.
4. Choose where it runs
Cluster just for this run (turns on, runs and turns off) is the default and the safest. On a cluster that is already on uses one of your clusters that is already on.
5. Parameters (optional) and save
In Parameters, one per line in the
name=valueformat. Click Save step.
If the notebook list is empty, the screen warns: "No published notebook yet. The pipeline only runs published versions of a notebook." Publish a version in the editor (Publish version) and come back.
The draft never runs in the pipeline. To take a notebook change to the pipeline, publish a new version and pick it in the step.
What the notebook finds ready#
When it runs as a step, the notebook already has these variables:
| Variable | What it contains |
|---|---|
spark | the Spark session, already connected to your datalake |
TABELA_ALVO | the full name of the step's target table |
CAMADA | silver or gold |
PARAMETROS | the step's parameters, as a text dictionary (PARAMETROS["name"]) |
RUN_ID | the pipeline run identifier |
RUN_INTERVAL_START, RUN_INTERVAL_END | the period requested when reprocessing (ISO date) or None |
Where the step writes#
The step writes to the target table, in one of these two ways:
- The notebook has a SQL cell. The last SQL cell becomes the target table: it replaces or appends, according to the chosen Load. Earlier SQL cells run as intermediate steps.
- The notebook is Python only. Your code writes to
TABELA_ALVO— for example, writing the final DataFrame with the write mode you want.
Example: a parameter uf_minima=10 in the step reaches the code as text.
minimo = int(PARAMETROS.get("uf_minima", "0"))
print(f"Gravando em {TABELA_ALVO} (camada {CAMADA}), mínimo {minimo}")The notebook can only write to your own workspace's datasets.
Conversion rules#
- Cells run in order; text cells are ignored.
%%sqlon the first line of a code cell makes it SQL (common in imported.ipynbfiles).- Interactive notebook commands (
%pip,!ls,%matplotlib) do not exist in a job: the step refuses and points to the cell. Libraries come from Extra Python libraries, on the cluster screen.
Where it runs and how much it costs#
| Where it runs | How it charges |
|---|---|
| Cluster just for this run (default) | a cluster is created for the run, used and deleted at the end, even if it fails. You pay for its lifetime (including the minutes to start), once per run — a retry does not charge twice |
| On a cluster that is already on | no separate charge: the running cluster already charges per hour |
Running cluster with an open notebook
If someone has a notebook open on the chosen cluster, the session holds the machines and the step may wait several minutes to start. For a scheduled pipeline, prefer the cluster just for the run.
The step's cost appears in the run's cost, in ETL/ELT → Runs.
Limits#
- Starting the cluster takes a few minutes; a notebook step makes the pipeline run longer than a SQL step.
- A file Bronze and a notebook step in the same run do not run together: run the Bronze and then the notebook's layer.