BwUniCluster2.0/Software/Python Dask: Difference between revisions
< BwUniCluster2.0 | Software
Jump to navigation
Jump to search
No edit summary |
|||
| Line 7: | Line 7: | ||
Please have a look at our [https://github.com/hpcraink/workshop-parallel-jupyter Workshop] on how to use Dask on bwUniCluster2.0 (2_Grundlagen: Environment erstellen and 6_Dask). This is currently only available in German. |
Please have a look at our [https://github.com/hpcraink/workshop-parallel-jupyter Workshop] on how to use Dask on bwUniCluster2.0 (2_Grundlagen: Environment erstellen and 6_Dask). This is currently only available in German. |
||
== Using Dask == |
|||
In a new interactive shell, execute the following commands in Python: |
|||
<pre> |
|||
>>> from dask_jobqueue import SLURMCluster |
|||
>>> cluster = SLURMCluster(cores=X, memory='X GB', queue='X') |
|||
</pre> |
|||
You have to specify how many cores and memory you want for one dask worker. Furthermore a [[BwUniCluster_2.0_Batch_Queues|batch queue]] is required. |
|||
<pre> |
|||
>>> cluster.scale (X) |
|||
</pre> |
|||
After executing this command with e.g. cluster.scale(5), dask will start to request five worker processes each with the specified amount of cores and memory. |
|||
With the command 'squeue' in bash you can check if your dask worker processes are actually running or if you have to wait until you get the requested resources. You have a better chance to get resources quickly if you additionally specify a walltime. You can check all options you have with the following command: |
|||
<pre> |
|||
>>> SLURMCluster? |
|||
</pre> |
|||
With the command |
|||
<pre> |
|||
>>> client |
|||
</pre> |
|||
dask will output all resources that are are actually available to distribute your computing. |
|||
== Dask Dashboard == |
== Dask Dashboard == |
||
Revision as of 10:50, 9 December 2021
This guide explains how to use Python Dask and dask-jobqueue on bwUniCluster2.0.
Installation and Usage
Please have a look at our Workshop on how to use Dask on bwUniCluster2.0 (2_Grundlagen: Environment erstellen and 6_Dask). This is currently only available in German.
Dask Dashboard
To forward the Dask Dashboard you have to do a ssh port forwarding with the machine on which you have started Dask.
$ ssh -N -L 8787:machineName:8787 yourusername@bwunicluster.scc.kit.edu
After executing this command you can access the dask dashboard in your local browser by typing 'localhost:8787/status'.