This page is an interface inventory: what is written here is the text that appears on the screen, literally.
Where it is: menu ETL/ELT → Clusters.
A cluster is the set of Spark machines where notebooks and transformations run. You only pay while it is on: turning it off deletes the machines and keeps the configuration, and turning it back on takes about 6 minutes. The cost comes out of your credit balance.
See the clusters (/clusters)#
Columns: Name · Status · Size · Cost per hour · Spent this month. To create one, click New cluster.
The status shows as Off, Starting, On, Stopping or Failed.
Create a cluster (/clusters/novo)#
- Give it a name in Cluster name.
- Under Size, choose Small (one machine), Medium (three machines), Large (five larger machines) or Custom (you choose Main machine, Worker machines and Disk per machine).
- Under Turn off automatically after being idle for, choose how long it waits idle before turning off. It turns off even if you close the page.
- Optional: Extra Python libraries (one per line, in pip format) and Spark settings (key=value).
- Check the Cost projection on the side: price per hour, 8 hours of use, working month and how much your balance covers. Click Create cluster.
Your plan defines how many clusters can be on at the same time and the maximum size of each one.
Turn on, turn off and follow the cost (/clusters/<id>)#
| You want to | Click |
|---|---|
| Start using it | Turn on (the status goes to Starting and then On) |
| Stop paying | Turn off |
| Delete the configuration | Delete (only with the cluster off) |
While on, the Live cost block shows On for, Price per hour, Spent so far, how much it would cost if it stayed idle until turning off by itself, and how many hours Your balance covers. To turn on, the balance must cover at least one hour; if the balance runs out while the cluster is on, it turns off by itself.
The Usage history lists each time the cluster was on, the duration, why it turned off and how much it cost.