BwUniCluster2.0/Software/Python Dask: Difference between revisions

From bwHPC Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
Line 19: Line 19:
>>> cluster.scale (X)
>>> cluster.scale (X)
</pre>
</pre>
After executing this command with e.g. cluster.scale(5), dask will start to request five worker processes each with the specified amount of cores and memory.


Replace
Replace

Revision as of 11:00, 19 April 2020

This guide explains how to use Python Dask and dask-jobqueue on bwUniCluster2.0.

Installation

Use on of our pre-configured Python modules and load them with 'module load ...'. You have to install the packages 'dask' and 'das-jobqueue' if your are using an own conda environment.

Using Dask

In a new interactive shell, execute the following commands in Python:

>>> from dask_jobqueue import SLURMCluster
>>> cluster = SLURMCluster(cores=X, memory='X GB', queue='X')

You have to specify how many cores and memory you want for one dask worker. Furthermore a batch queue is required.

>>> cluster.scale (X)

After executing this command with e.g. cluster.scale(5), dask will start to request five worker processes each with the specified amount of cores and memory.

Replace