When the transformation goes beyond what fits in SQL — cleaning text, applying a forecasting model, crossing large files, using a Python library — you write a notebook and run it on a Spark cluster in your workspace.
The two concepts#
| What it is | You pay | |
|---|---|---|
| Cluster | the set of Spark machines where the code runs. You choose the size, turn it on when you will use it and it turns off automatically after being idle for a while | only while it is on, per hour, from the credit balance |
| Notebook | a sequence of cells — Python / PySpark, SQL or Text (Markdown) — that you write and run cell by cell, seeing the result right away | nothing beyond the cluster where it runs |
The code is regular Spark: a notebook from here runs the same on any PySpark,
and you can import and download notebooks in the .ipynb (Jupyter)
format.
The notebook reads and writes to the layers of your datalake (Bronze, Silver and Gold). The cluster only sees your workspace's data — no one else's.
When to use a notebook, and when not to#
| You want to | Use |
|---|---|
| Join, filter and aggregate tables | a guided or free SQL Silver/Gold step — simpler and cheaper |
| Explore data, test an idea, investigate a number | a notebook on a running cluster |
| A Python transformation (library, model, rule that is hard in SQL) that runs every day | a published notebook as a pipeline step |
Step by step#
1. Create a cluster
In ETL/ELT → Clusters, click New cluster. Give it a name, choose the Size (Small is enough to start) and the Turn off automatically after being idle for time. Check the Cost projection and click Create cluster.
2. Turn the cluster on
On the cluster detail, click Turn on. It goes through Starting and is On in about 6 minutes.
3. Create a notebook
In ETL/ELT → Notebooks, click New notebook and then Create notebook (it comes with pandas, PySpark and SQL examples) — or choose an
.ipynbfile and click Import notebook.4. Attach and run
In the editor, click Attach to a cluster, choose the cluster and run the cells (▶ or Shift+Enter). Use Run all cells for the whole notebook.
5. Publish a version
When the notebook is ready to run on its own, click Publish version. Only published versions go into a pipeline.
6. Turn the cluster off
Done? Click Turn off on the cluster. If you forget, it turns off by itself after the time you chose.
Cost#
- The cluster charges per hour on, by the chosen size. The creation screen shows the Cost projection (per hour, 8 hours of use, working month and how much your balance covers).
- To turn it on, the balance must cover at least one hour. If the balance runs out with the cluster on, it turns off by itself.
- Detaching a notebook does not turn the cluster off. What stops the charge is Turn off (or the automatic shutdown).
- Your plan defines how many clusters can be on at the same time and the maximum size of each.
- A pipeline notebook step, by default, starts a cluster just for that run and deletes it at the end — you pay only for the minutes of the run.
Libraries#
- This session only:
%pip install packagein a Python cell. - Permanently and on all machines (required for distributed functions, UDFs, and for the pipeline step): add the library to Extra Python libraries on the cluster screen.
Limits#
- Terminal commands (
!command,%%bash,%%sh,%system) do not run: the screen refuses with Command not allowed. - The SQL cell shows up to 1,000 result rows.
- Turning on a cluster takes a few minutes; for immediate answers, use the data Console or a query.