This page is an interface inventory: what is written here is the text that appears on screen, literally.
Where it is: menu ETL/ELT → Notebooks.
A notebook is a sequence of cells that run on your Spark cluster: plain Python (pandas, for example), PySpark and SQL. Use it to explore data, test transformations and create new tables. The code is regular Spark: it runs the same on any PySpark, with nothing proprietary.
View notebooks (/notebooks)#
Notebooks are grouped by folder; those without a folder are under No folder. Each one shows its published version or Draft. To create one, click New notebook.
Create or import (/notebooks/novo)#
- Give it a name and, if you want, a Folder (optional) (e.g.
Sales/Monthly). - To start from scratch, click Create notebook: it comes with examples of pandas, PySpark reading your Silver layer, and SQL.
- To bring a notebook from elsewhere, choose the
.ipynbfile and click Import notebook. Outputs are not imported; cells run again on your cluster.
Write and run (/notebooks/<id>)#
At the top right, click Attach to a cluster and choose where the cells will run (each one shows its state and hourly cost). If the cluster is stopped, use turn on and attach — starting takes a few minutes. Once attached, the cluster shows as a chip with name, state and R$/hour; click it to Detach, switch cluster, Turn on the cluster or turn it off, or Restart the session. Detaching does not turn the cluster off: it keeps billing until you turn it off.
The icon bar (hover to see each name):
| You want to | Do |
|---|---|
| Run a cell | ▶ next to the cell, or Shift+Enter (moves to the next one) |
| Run the whole notebook | Run all cells |
| Stop what is running | Interrupt |
| Start the session from scratch (clears variables) | Restart the session |
| Take the notebook to another tool | Download the notebook (.ipynb) |
| Freeze the version that pipelines will run | Publish version |
Each cell has a Cell name (shown on top of it and in the pipeline program) and the Jupyter keyboard shortcuts: Esc leaves editing, A/B add a cell above/below, D D deletes, M turns it into text, Y into code.
Each cell has a type: Python / PySpark, SQL or Text (Markdown). The SQL cell runs on Spark and shows the result as a table (up to 1,000 rows). The draft saves itself on every change.
Libraries
- This session only: in a Python cell, run
%pip install package. It applies only to the open session and only to the cluster's main machine. - On all machines (needed to use it inside distributed Spark functions, UDFs): add the library to Extra Python libraries on the cluster screen.
What does not run
Terminal commands do not run on the cluster: lines starting with ! and
%%bash, %%sh or %system cells are refused with the Command not
allowed notice. Use Python for the same work.