Difference between revisions of "BwUniCluster2.0/Software/Python Dask"
< BwUniCluster2.0 | Software
H Winkhardt (talk | contribs) m (H Winkhardt moved page BwUniCluster 2.0 Python Dask to BwUniCluster2.0/Software/Python Dask without leaving a redirect) |
|||
| (9 intermediate revisions by 2 users not shown) | |||
| Line 4: | Line 4: | ||
This guide explains how to use Python Dask and dask-jobqueue on bwUniCluster2.0. |
This guide explains how to use Python Dask and dask-jobqueue on bwUniCluster2.0. |
||
| − | == Installation == |
+ | == Installation and Usage == |
| + | Please have a look at our [https://github.com/hpcraink/workshop-parallel-jupyter Workshop] on how to use Dask on bwUniCluster2.0 (2_Grundlagen: Environment erstellen and 6_Dask). This is currently only available in German. |
||
| − | Use on of our pre-configured Python modules and load them with 'module load ...'. You have to install the packages 'dask' and 'dask-jobqueue' if your are using an own conda environment. |
||
| − | |||
| − | == Using Dask == |
||
| − | In a new interactive shell, execute the following commands in Python: |
||
| − | |||
| − | <pre> |
||
| − | >>> from dask_jobqueue import SLURMCluster |
||
| − | >>> cluster = SLURMCluster(cores=X, memory='X GB', queue='X') |
||
| − | </pre> |
||
| − | You have to specify how many cores and memory you want for one dask worker. Furthermore a [[BwUniCluster_2.0_Batch_Queues|batch queue]] is required. |
||
| − | |||
| − | <pre> |
||
| − | >>> cluster.scale (X) |
||
| − | </pre> |
||
| − | After executing this command with e.g. cluster.scale(5), dask will start to request five worker processes each with the specified amount of cores and memory. |
||
| − | |||
| − | With the command 'squeue' in bash you can check if your dask worker processes are actually running or if you have to wait until you get the requested resources. You have a better chance to get resources quickly if you additionally specify a walltime. You can check all options you have with the following command: |
||
| − | |||
| − | <pre> |
||
| − | >>> SLURMCluster? |
||
| − | </pre> |
||
| − | |||
| − | With the command |
||
| − | |||
| − | <pre> |
||
| − | >>> client |
||
| − | </pre> |
||
| − | |||
| − | dask will output all resources that are are actually available to distribute your computing. |
||
| − | |||
| − | == Dask Dashboard == |
||
| − | To forward the Dask Dashboard you have to do a ssh port forwarding with the machine on which you have started Dask. |
||
| − | |||
| − | <pre> |
||
| − | $ ssh -N -L 8787:machineName:8787 yourusername@bwunicluster.scc.kit.edu |
||
| − | </pre> |
||
| − | |||
| − | After executing this command you can access the dask dashboard in your local browser by typing 'localhost:8787/status'. |
||
| − | |||
| − | |||
| − | ---- |
||
| − | [[Category:bwUniCluster_2.0|Access]][[Category:Access|bwUniCluster 2.0]] |
||
Latest revision as of 11:46, 27 February 2024
This guide explains how to use Python Dask and dask-jobqueue on bwUniCluster2.0.
Installation and Usage
Please have a look at our Workshop on how to use Dask on bwUniCluster2.0 (2_Grundlagen: Environment erstellen and 6_Dask). This is currently only available in German.
