JUSTUS2/Software/Julia/Parallel Programming: Difference between revisions
M Carmesin (talk | contribs) No edit summary |
M Carmesin (talk | contribs) No edit summary |
||
| Line 1: | Line 1: | ||
= Parallel Programming in Julia = |
= Parallel Programming in Julia = |
||
Julia supports several paradigms of parallel programming |
Julia supports several paradigms of parallel programming: |
||
# Implicit multi-threading by math libraries (OpenBLAS, MKL) |
# Implicit multi-threading by math libraries (OpenBLAS, MKL) |
||
# Explicit multi-threading using Julia threads (e.g. `Threads.@threads for`) |
# Explicit multi-threading using Julia threads (e.g. `Threads.@threads for`) |
||
# Multiple processes on one ore more nodes |
# Multiple processes on one ore more nodes |
||
#* <code>Distributed</code> package and <code>SlurmManager</code> from [https://github.com/JuliaParallel/ClusterManagers.jl <code>ClusterManagers.jl</code>] package, <code>@distributed for</code>-loops |
#* <code>Distributed.jl</code> package and <code>SlurmManager</code> from [https://github.com/JuliaParallel/ClusterManagers.jl <code>ClusterManagers.jl</code>] package, (e.g.<code>@distributed for</code>-loops) |
||
#* [https://github.com/JuliaParallel/MPI.jl <code>MPI.jl</code>] |
#* [https://github.com/JuliaParallel/MPI.jl <code>MPI.jl</code>] |
||
# Execution on GPUs/CUDA using [https://cuda.juliagpu.org/stable/ <code>CUDA.jl</code> ] |
# Execution on GPUs/CUDA using [https://cuda.juliagpu.org/stable/ <code>CUDA.jl</code> ] |
||
All paradigms may be used at the same time, but must be chosen carefully, to obtain the desired performance. |
|||
== Implict Multi-Threading == |
|||
The number of threads used by the mathematical linear algebra libraries may be configured using <code>BLAS.set_num_threads()</code> from the <code>LinearAlgebra</code> package. Alternatively you can set the environment variables <code>OPENBLAS_NUM_THREADS</code> or <code>MKL_NUM_THREADS</code> if you use MKL. |
|||
If your code is already multi-threaded, you probably want to set the number of BLAS threads to 1, in order to avoid running too many competing threads, as every Julia thread comes with its own BLAS threads. |
|||
== Explicit Multi-Threading == |
|||
Start Julia with option <code>-t x</code> where x is |
|||
the number of (Julia) threads or the keyword <code>auto</code>, which however doesn't determine correctly the number of threads requested from SLURM with the option <code>--cpus-per-task</code>. Alternatively, you can set environment variable <code>JULIA_NUM_THREADS</code>. See the [https://docs.julialang.org/en/v1/manual/multi-threading/ Julia documentation] for more details. |
|||
Revision as of 17:52, 8 October 2024
Parallel Programming in Julia
Julia supports several paradigms of parallel programming:
- Implicit multi-threading by math libraries (OpenBLAS, MKL)
- Explicit multi-threading using Julia threads (e.g. `Threads.@threads for`)
- Multiple processes on one ore more nodes
Distributed.jlpackage andSlurmManagerfromClusterManagers.jlpackage, (e.g.@distributed for-loops)MPI.jl
- Execution on GPUs/CUDA using
CUDA.jl
All paradigms may be used at the same time, but must be chosen carefully, to obtain the desired performance.
Implict Multi-Threading
The number of threads used by the mathematical linear algebra libraries may be configured using BLAS.set_num_threads() from the LinearAlgebra package. Alternatively you can set the environment variables OPENBLAS_NUM_THREADS or MKL_NUM_THREADS if you use MKL.
If your code is already multi-threaded, you probably want to set the number of BLAS threads to 1, in order to avoid running too many competing threads, as every Julia thread comes with its own BLAS threads.
Explicit Multi-Threading
Start Julia with option -t x where x is
the number of (Julia) threads or the keyword auto, which however doesn't determine correctly the number of threads requested from SLURM with the option --cpus-per-task. Alternatively, you can set environment variable JULIA_NUM_THREADS. See the Julia documentation for more details.