Skip to content

Automatic translation — the Portuguese version is the reference. View in Portuguese

Updated on 2026-10-05

Spark notebooks and clusters — what they are and how to use them

The concept of cluster and notebook, when to use each, how much they cost, and the step by step to create a cluster, write a notebook and take it to the pipeline.

On this page (7)

When the transformation goes beyond what fits in SQL — cleaning text, applying a forecasting model, crossing large files, using a Python library — you write a notebook and run it on a Spark cluster in your workspace.

The two concepts#

What it isYou pay
Clusterthe set of Spark machines where the code runs. You choose the size, turn it on when you will use it and it turns off automatically after being idle for a whileonly while it is on, per hour, from the credit balance
Notebooka sequence of cells — Python / PySpark, SQL or Text (Markdown) — that you write and run cell by cell, seeing the result right awaynothing beyond the cluster where it runs

The code is regular Spark: a notebook from here runs the same on any PySpark, and you can import and download notebooks in the .ipynb (Jupyter) format.

The notebook reads and writes to the layers of your datalake (Bronze, Silver and Gold). The cluster only sees your workspace's data — no one else's.

When to use a notebook, and when not to#

You want toUse
Join, filter and aggregate tablesa guided or free SQL Silver/Gold step — simpler and cheaper
Explore data, test an idea, investigate a numbera notebook on a running cluster
A Python transformation (library, model, rule that is hard in SQL) that runs every daya published notebook as a pipeline step

Step by step#

  1. 1. Create a cluster

    In ETL/ELT → Clusters, click New cluster. Give it a name, choose the Size (Small is enough to start) and the Turn off automatically after being idle for time. Check the Cost projection and click Create cluster.

  2. 2. Turn the cluster on

    On the cluster detail, click Turn on. It goes through Starting and is On in about 6 minutes.

  3. 3. Create a notebook

    In ETL/ELT → Notebooks, click New notebook and then Create notebook (it comes with pandas, PySpark and SQL examples) — or choose an .ipynb file and click Import notebook.

  4. 4. Attach and run

    In the editor, click Attach to a cluster, choose the cluster and run the cells (▶ or Shift+Enter). Use Run all cells for the whole notebook.

  5. 5. Publish a version

    When the notebook is ready to run on its own, click Publish version. Only published versions go into a pipeline.

  6. 6. Turn the cluster off

    Done? Click Turn off on the cluster. If you forget, it turns off by itself after the time you chose.

Cluster list with Name, Status, Size, Cost per hour and Spent this month, and the New cluster button
Notebook editor with text, Python / PySpark and SQL cells, the query result as a table and, at the top, the chosen cluster and the Publish version button

Cost#

  • The cluster charges per hour on, by the chosen size. The creation screen shows the Cost projection (per hour, 8 hours of use, working month and how much your balance covers).
  • To turn it on, the balance must cover at least one hour. If the balance runs out with the cluster on, it turns off by itself.
  • Detaching a notebook does not turn the cluster off. What stops the charge is Turn off (or the automatic shutdown).
  • Your plan defines how many clusters can be on at the same time and the maximum size of each.
  • A pipeline notebook step, by default, starts a cluster just for that run and deletes it at the end — you pay only for the minutes of the run.

Libraries#

  • This session only: %pip install package in a Python cell.
  • Permanently and on all machines (required for distributed functions, UDFs, and for the pipeline step): add the library to Extra Python libraries on the cluster screen.

Limits#

  • Terminal commands (!command, %%bash, %%sh, %system) do not run: the screen refuses with Command not allowed.
  • The SQL cell shows up to 1,000 result rows.
  • Turning on a cluster takes a few minutes; for immediate answers, use the data Console or a query.

Read more#

Related links