Skip to content

Automatic translation — the Portuguese version is the reference. View in Portuguese

Updated on 2026-10-05

Notebook step in Silver or Gold — run a notebook inside the pipeline

How to add a published notebook as a Silver or Gold pipeline step, what the code finds ready (spark, TABELA_ALVO, PARAMETROS), where the step writes and where it runs.

On this page (7)

Besides guided and free SQL steps, a Silver or Gold table can be produced by a notebook. The pipeline runs the notebook's published version as a Spark job, in order with the other steps, every time the pipeline runs — on schedule, through Run now or when reprocessing a period.

First: create and publish the notebook. See Spark notebooks and clusters.

Create the step#

  1. 1. Open the layer

    On the pipeline detail, go to the Silver (or Gold) layer. Next to New Silver (or New Gold), click Notebook.

  2. 2. Choose the notebook

    In Published notebook, choose the notebook and, in Version, the version the pipeline will run (the most recent comes selected).

  3. 3. Say where to write

    In Target table, name the table the step produces. In Load, choose Replace the table or Append rows.

  4. 4. Choose where it runs

    Cluster just for this run (turns on, runs and turns off) is the default and the safest. On a cluster that is already on uses one of your clusters that is already on.

  5. 5. Parameters (optional) and save

    In Parameters, one per line in the name=value format. Click Save step.

If the notebook list is empty, the screen warns: "No published notebook yet. The pipeline only runs published versions of a notebook." Publish a version in the editor (Publish version) and come back.

The draft never runs in the pipeline. To take a notebook change to the pipeline, publish a new version and pick it in the step.

What the notebook finds ready#

When it runs as a step, the notebook already has these variables:

VariableWhat it contains
sparkthe Spark session, already connected to your datalake
TABELA_ALVOthe full name of the step's target table
CAMADAsilver or gold
PARAMETROSthe step's parameters, as a text dictionary (PARAMETROS["name"])
RUN_IDthe pipeline run identifier
RUN_INTERVAL_START, RUN_INTERVAL_ENDthe period requested when reprocessing (ISO date) or None

Where the step writes#

The step writes to the target table, in one of these two ways:

  1. The notebook has a SQL cell. The last SQL cell becomes the target table: it replaces or appends, according to the chosen Load. Earlier SQL cells run as intermediate steps.
  2. The notebook is Python only. Your code writes to TABELA_ALVO — for example, writing the final DataFrame with the write mode you want.

Example: a parameter uf_minima=10 in the step reaches the code as text.

python
minimo = int(PARAMETROS.get("uf_minima", "0"))
print(f"Gravando em {TABELA_ALVO} (camada {CAMADA}), mínimo {minimo}")

The notebook can only write to your own workspace's datasets.

Conversion rules#

  • Cells run in order; text cells are ignored.
  • %%sql on the first line of a code cell makes it SQL (common in imported .ipynb files).
  • Interactive notebook commands (%pip, !ls, %matplotlib) do not exist in a job: the step refuses and points to the cell. Libraries come from Extra Python libraries, on the cluster screen.

Where it runs and how much it costs#

Where it runsHow it charges
Cluster just for this run (default)a cluster is created for the run, used and deleted at the end, even if it fails. You pay for its lifetime (including the minutes to start), once per run — a retry does not charge twice
On a cluster that is already onno separate charge: the running cluster already charges per hour

Running cluster with an open notebook

If someone has a notebook open on the chosen cluster, the session holds the machines and the step may wait several minutes to start. For a scheduled pipeline, prefer the cluster just for the run.

The step's cost appears in the run's cost, in ETL/ELT → Runs.

Limits#

  • Starting the cluster takes a few minutes; a notebook step makes the pipeline run longer than a SQL step.
  • A file Bronze and a notebook step in the same run do not run together: run the Bronze and then the notebook's layer.

Related#

Related links