M Carmesin: Created page with "= Parallel Programming in Julia = Julia supports several paradigms of parallel programming: # Implicit multi-threading by math libraries (OpenBLAS, MKL, AMD AOCL) # Explicit multi-threading using Julia threads (e.g. `Threads.@threads for`) or [https://github.com/JuliaSIMD/Polyester.jl Polyester.jl ] # Multiple processes on one ore more nodes #* `Distributed.jl` package and `SlurmManager` from [https://github.com/JuliaParallel/SlurmClusterManager.j..."

2026-11-11T10:30:26Z

Created page with "= Parallel Programming in Julia = Julia supports several paradigms of parallel programming: # Implicit multi-threading by math libraries (OpenBLAS, MKL, AMD AOCL) # Explicit multi-threading using Julia threads (e.g. `Threads.@threads for`) or [https://github.com/JuliaSIMD/Polyester.jl Polyester.jl ] # Multiple processes on one ore more nodes #* <code>Distributed.jl</code> package and <code>SlurmManager</code> from [https://github.com/JuliaParallel/SlurmClusterManager.j..."

New page

= Parallel Programming in Julia =

Julia supports several paradigms of parallel programming:

# Implicit multi-threading by math libraries (OpenBLAS, MKL, AMD AOCL)
# Explicit multi-threading using Julia threads (e.g. `Threads.@threads for`) or [https://github.com/JuliaSIMD/Polyester.jl Polyester.jl ]
# Multiple processes on one ore more nodes
#* <code>Distributed.jl</code> package and <code>SlurmManager</code> from [https://github.com/JuliaParallel/SlurmClusterManager.jl <code>SlurmClusterManager.jl</code>] package, (e.g.<code>@distributed for</code>-loops)
#* [https://github.com/JuliaParallel/MPI.jl <code>MPI.jl</code>]
# Execution on GPUs/CUDA using [https://cuda.juliagpu.org/stable/ <code>CUDA.jl</code> ]

All paradigms may be used at the same time, but must be chosen carefully, to obtain the desired performance.

== Implict Multi-Threading ==

The number of threads used by the mathematical linear algebra libraries may be configured using <code>BLAS.set_num_threads()</code> from the <code>LinearAlgebra</code> package. Alternatively you can set the environment variables <code>OPENBLAS_NUM_THREADS</code> or <code>MKL_NUM_THREADS</code> if you use MKL or <code>BLIS_NUM_THREADS</code> if you use AMD BLIS.

If your code is already multi-threaded, you probably want to set the number of BLAS threads to 1, in order to avoid running too many competing threads, as every Julia thread comes with its own BLAS threads.

== Explicit Multi-Threading ==
Start Julia with option <code>-t x</code> where x is
the number of (Julia) threads or the keyword <code>auto</code>, which however doesn't determine correctly the number of threads requested from SLURM with the option <code>--cpus-per-task</code>. Alternatively, you can set the environment variable <code>JULIA_NUM_THREADS</code>. See the [https://docs.julialang.org/en/v1/manual/multi-threading/ Julia documentation] for more details.

== Multiple Processes ==
With the [https://docs.julialang.org/en/v1/manual/distributed-computing/ Distributed package] Julia has native support for distributed computing using multiple processes on different nodes. To integrate well into SLURM, the use of the [https://github.com/JuliaParallel/ClusterManagers.jl <code>ClusterManagers.jl</code>], providing the <code>addprocs_slurm()</code> function, is advised to spawn the worker processes.

== MPI ==

Distributed computing using MPI can be performed leveraging the [https://github.com/JuliaParallel/MPI.jl <code>MPI.jl</code>] package, which provides Julia wrappers for most of the standard MPI functions.

== CUDA ==
Julia supports computations on NVidia GPUS using the [https://cuda.juliagpu.org/stable/ CUDA.jl] package. It provides the possibility to write own kernels as well as wrappers for libraries like cuBLAS or cuFFT, that contain implementations of standard numerical routines optimized for GPUs.

== Higher Level Packages ==

There are several Julia packages that allow for mixing/changing the different paradigms for parallel computing with minimal code changes:

* [https://github.com/JuliaFolds2/FLoops.jl <code>FLoops.jl</code>] and its backend [https://juliafolds2.github.io/Folds.jl/dev/ <code>Folds.jl</code>]
* [https://juliaparallel.org/Dagger.jl/stable/ <code>Dagger.jl</code>]

Justus3/Software/Julia/Parallel Programming - Revision history