{{Softwarepage}}

{| width=760px class="wikitable"
|-
! Description !! Content
|-
| module load
| chem/dalton
|-
| License
| Open-source software, distributed under the GNU Lesser General Public License (LGPL). [[#License|More...]]
|-
| Citing
| [https://daltonprogram.org/citation/ Publications]
|-
| Links
| [http://www.daltonprogram.org/index.html Homepage] | [https://daltonprogram.org/documentation/ Documentation]
|-
| Graphical Interface
| No
|}
= Description =

'''Dalton''' (named after [https://en.wikipedia.org/wiki/John_Dalton John Dalton]) is an ab initio quantum chemistry computer program designed to to allow convenient, automated determination of a large number of molecular properties based on an HF, HF-srDFT, DFT, MP2, CC, CI, MCSCF, or MC-srDFT reference wave function. For additional information on features please visit the [https://daltonprogram.org/features/ Description of the Dalton suite features] web page.

= License =

Dalton is free, open-source software released under the GNU Lesser General Public License (LGPL). Anyone interested in using the Dalton program suite must read the conditions described in its [https://gitlab.com/dalton/dalton/-/blob/master/LICENSE license agreement].

= Usage =
== Hints for using Dalton ==

Scratch files are written to $SCRATCH by default. This configuration option can be changed by setting the environment variable $DALTON_TMPDIR (e.g., to a dedicated [[workspace]]) before starting your calculations with Dalton.

= FAQ =

'''Q:''' What to do if my simulations abort with MEMGET ERROR, insufficient work space in memory ?

'''A:''' Increase Dalton's usable work memory with either -mb or -gb on the command line.

= Useful links =

* [https://daltonprogram.org/manuals/dalton2020manual.pdf Documentation (english)]
* [http://forum.daltonprogram.org Forum (english)]
* [https://en.wikipedia.org/wiki/Dalton_(program) Wikipedia article (english)]
* [https://daltonprogram.org/tools/ Plugins for Dalton (english)]
----
[[Category:Chemistry software]][[Category:BwForCluster_Chemistry]][[Category:BwForCluster_JUSTUS_2]]

BwUniCluster2.0/Software/R/Glmnet

2021-11-18T17:11:48Z

F Wagner: /* Installation instructions */

= Installation instructions =

Consider starting an interactive job for compiling. Copy and paste the following to your shell.

<pre>
# Load the R software module, e.g.
module load math/R/4.1.2

# Prepare .R directory (if it does not already exists)
mkdir -p ~/.R

# Write the following environment variables to Makevars
echo "CXX14=icpc" >> ~/.R/Makevars
echo "CXX14FLAGS=-O3 -fPIC -std=c++14 -axCORE-AVX512,CORE-AVX2,AVX -xSSE4.2 -fp-model strict -qopenmp" >> ~/.R/Makevars

# Install the glmnet package from within R session
R -q
> install.packages("glmnet", dependencies=TRUE)

# Run a quick test
> library(glmnet)
> data(QuickStartExample)
> x <- QuickStartExample$x
> y <- QuickStartExample$y
> fit <- glmnet(x, y)
> print(fit)
</pre>

BwUniCluster2.0/Software/R/Glmnet

2021-11-18T17:09:10Z

F Wagner: Created page with "= Installation instructions = Consider starting an interactive job for compiling. Copy and paste the following to your shell. <pre> # Load the R software module, e.g. module..."

BwUniCluster2.0/Software/R

2021-11-18T17:05:36Z

F Wagner: /* Optional packages for R */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| math/R
|-
| License
| GPL
|-
| Citing
| n/a
|-
| Links
| [http://www.r-project.org/ Homepage] | [http://cran.r-project.org/manuals.html Documentation]
|-
| Graphical Interface
| No
|-
| Plugins
| User dependent
|}

= Description =

'''R''' is a language and environment for statistical computing and graphics. It is a GNU project which is similar to the S language and environment which was developed at Bell Laboratories (formerly AT&T, now Lucent Technologies) by John Chambers and colleagues. R can be considered as a different implementation of S. There are some important differences, but much code written for S runs unaltered under R.

R provides a wide variety of statistical (linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering, …) and graphical techniques, and is highly extensible. The S language is often the vehicle of choice for research in statistical methodology, and R provides an Open Source route to participation in that activity.

One of R’s strengths is the ease with which well-designed publication-quality plots can be produced, including mathematical symbols and formulae where needed. Great care has been taken over the defaults for the minor design choices in graphics, but the user retains full control.

R is available as Free Software under the terms of the Free Software Foundation’s GNU General Public License in source code form. It compiles and runs on a wide variety of UNIX platforms and similar systems (including FreeBSD and Linux), Windows and MacOS.

= Availability =

R is available on selected bwHPC-Clusters. A complete list of versions currently installed on the bwHPC-Clusters can be obtained from the [https://www.bwhpc.de/software.html Cluster Information System (CIS)].

In order to check which versions of R are installed on the compute cluster, run the following command:
<pre>
$ module avail math/R
</pre>

= Usage =
The R installation also provides the standalone library libRmath. This library allows you to access R routines from your own C or C++ programs (see section 9 of the 'R Installation and Administration' manual).

== Loading the module ==

You can load the default version of R with the following command:
<pre>
$ module load math/R
</pre>

The module will try to load all modules it needs to function (e.g. compiler/intel). If loading the module fails, check if you have already loaded one of those modules, but not in the version required by R.

If you wish to load another (older) version of R, you can do so using
<pre>
$ module load math/R/<version>
</pre>
with <version> specifying the desired version.

== Program Binaries ==
Standard usage:

<pre>
Usage: R [options] [< infile] [> outfile]
R CMD command [arguments]

Example: R CMD BATCH script.R
</pre>

Executing R in batch mode:

<pre>
R CMD BATCH --no-save --no-restore <INPUT_FILE>.R
</pre>

For help run
<pre>
R --help
</pre>

For command help run
<pre>
R CMD command --help
</pre>

Further information and help
<pre>
Man pages: man R man Rscript
Info pages, e.g.: info R-intro info R-FAQ
Manuals: $R_DOC_DIR/manual
</pre>

== Multithreading in R ==

An easy way to use multiple cores on a single node with R is to use the [https://cran.r-project.org/web/packages/doParallel/vignettes/gettingstartedParallel.pdf doParallel] package in combination with [https://cran.r-project.org/web/packages/foreach/vignettes/foreach.pdf foreach].

= Examples =
As with all processes that require more than a few minutes to run, non-trivial compute jobs must be submitted to the cluster queuing system.

Example scripts are available in the directory $R_EXA_DIR:
<pre>
$ module show math/R # show environment variables, which will be available after 'module load'
$ module load math/R # load module
$ ls $R_EXA_DIR # show content of directory $R_EXA_DIR
$ cat $R_EXA_DIR/README # show examples README
</pre>



= Installing R-Packages into your home folder =
Since we cannot provide a software module for every R package, we recommend to install special R packages locally into your home folder. One possibility doing this is from within an interactive R session:

<pre>
> library() # List preinstalled packages
> install.packages('package_name', repos="http://cran.r-project.org") # Installing your R package and the dependencies
> library(package_name) # Loading the package into you R instance
</pre>

The package is now installed permanently in your home folder and is available every time you start R.

'''Note:'''

By default R uses a version (and platform) specific path for personal libraries, such as
"$HOME/R/x86_64-pc-linux-gnu-library/x.y" for R version x.y.z. This directory will be created automatically (after confirmation) when installing a personal package for the first time.

Users can customize a common location of their personal library packages, e.g. ~/R_libs, rather than the default location. A customized directory must exist before installing a personal package for the first time, i.e.

<pre>
$ mkdir -p ~/R_libs
</pre>

The location must also be defined in a configuration file ~/.Renviron within the home directory containing the following line:

<pre>
R_LIBS_USER="~/R_libs"
</pre>

By setting up a (fixed) custom location for personal library packages, any personal package installed into that directory will be visible across different R versions. This may be advantageous if the packages are to be used with different (future) R versions.

A version specific path, such as the default path, allows users to maintain multiple personal library stacks for different (major and minor) R versions and does also prevent users from mixing their stack with libraries built with different R versions.

The drawback is that, whenever switching to a new R release, the personal library stack '''must''' be rebuilt with that new R version into the corresponding (version specific) library path. This is considered good practice anyway in order to ensure a consistent personal library stack for any specific R version in use. Mixing libraries built with different major and minor R versions is discouraged, as this may (or may not) result in unpredictable and subtle errors. Packages that are built and installed with one version of R may be incompatible with a newer version of R, at least when the major or minor version changes. The same is true if several versions are used simultaneously, e.g. a newer R version for a more recently started project and and older version for another project (but eventually picking up libraries built with the newer R version).

Special care has also to be taken by users who always load the default version, i.e.

<pre>
$ module load math/R
</pre>

as the default version number may change any time. Is is therefore highly recommended to always load a specific version, e.g.

<pre>
$ module load math/R/3.6.3
</pre>

== Optional packages for R ==

The following guides provide detailed instructions about how to build optional R packages. Please write a [http://www.support.bwhpc-c5.de ticket] if the instructions do not work for you or are outdated.

* [[Rgdal]]
* [[Rjags]]
* [[Rstan]]
* [[glmnet]]

= Version-specific Information =

For specific help about a particular R version, check the information available via the module system with the following command:
<pre>
$ module help math/R
</pre>

= Installed R plugins =
* [[CummeRbund_(R-package)|Bioinformatics: cummeRbund]]
 
 

----
[[Category:Mathematics software]]
[[Category:BwUniCluster]]
[[Category:BwUniCluster_2.0]]
[[Category:BwForCluster_BinAC]]
[[Category:BwForCluster_MLS&WISO_Production]]

JUSTUS2/Software/Singularity

2021-10-11T11:19:44Z

F Wagner: /* NGC environment modules on JUSTUS 2 */

{| width=760px class="wikitable"
|-
! Description !! Content
|-
| module load
| system/singularity
|-
| Availability
| [[BwForCluster_JUSTUS_2]]
|-
| License
| Open-source software, distributed under the 3-clause BSD License. [[#License|More...]]
|-
| Citing
| ---
|-
| Links
| [https://sylabs.io/singularity/ Homepage] | [https://sylabs.io/docs/ Documentation]
|-
| Graphical Interface
| No
|}
= Description =

'''Singularity''' is a container platform.

= License =

Singularity is free, open-source software released under the 3-clause BSD license. Please read the [https://sylabs.io/guides/3.7/user-guide/license.html license] for additional information about Singularity.

= Usage =

== Loading the module ==

You can load the default version of Singularity with the following command:
<pre>
$ module load system/singularity
</pre>

If you wish to load another (older) version of Singularity, you can do so using
<pre>
$ module load system/singularity/<version>
</pre>
with <version> specifying the desired version.

Important: On JUSTUS 2, Singularity is available on all compute nodes. You do not have to load a module.

== Program Binaries ==

The binary ''singularity'' is main program of the container platform.

To get help using Singularity execute the following command:
<pre>
$ singularity --help
</pre>

Furthermore, a man page is available and can be accessed by typing:
<pre>
$ man singularity
</pre>

For additional information about how to use Singularity, please consult the [https://sylabs.io/docs/ documentation].

Important: On JUSTUS 2, Singularity is only available on compute nodes. You must switch to such a node to invoke any ''singularity'' commands.

== Batch jobs with containers ==

Batch jobs utilizing Singularity containers are generally built the same way as all other [https://wiki.bwhpc.de/e/BwForCluster_JUSTUS_2_Slurm_HOWTO#How_to_submit_a_batch_job.3F batch jobs], where the job script contains ''singularity'' commands. For example:

<pre>
#!/bin/bash
# Allocate one node
#SBATCH --nodes=1
# Number of program instances to be executed
#SBATCH --tasks-per-node=4
# 8 GB memory required per node
#SBATCH --mem=16G
# Maximum run time of job
#SBATCH --time=1:00:00
# Give job a reasonable name
#SBATCH --job-name=Singularity
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=singularity_job-%j.out
# File name for error output
#SBATCH --error=singularity_job-%j.err

module load your/singularity/version #(not needed on JUSTUS 2, but could be necessary on other system)

cd your/workspace

# Run container (two options to start a container)
singularity run [options] <container>
singularity exec [options] <container> <command>
</pre>

Keep in mind that other modules you may have loaded will not be available inside the container.

=== Using GPUs ===

<pre>
#!/bin/bash
[…]
# Allocate one GPU per node
#SBATCH --partition=gpu
#SBATCH --gres=gpu:1
[…]

module load your/singularity/version #(not needed on JUSTUS 2, but could be necessary on other system)

cd your/workspace

# Run container (two options to start a container)
singularity run --nv [options] <container>
singularity exec --nv [options] <container> <command>
</pre>

Using the flag is advisable, but may be omitted if the correct GPU- and driver-APIs are available on the container.

= Examples =

== Run your first container on JUSTUS 2 ==

Build a TensorFlow container with Singularity and execute a Python command:
<pre>
# request interactive node with GPUs
$ srun --nodes=1 --exclusive --gres=gpu:2 --pty bash

# create workspace and navigate into it
$ WORKSPACE=`ws_allocate tensorflow 3`
$ cd $WORKSPACE

# build container
$ singularity build tensorflow-20.11-tf2-py3.sif docker://nvcr.io/nvidia/tensorflow:20.11-tf2-py3

# execute Python command
$ singularity exec --nv tensorflow-20.11-tf2-py3.sif python -c 'import tensorflow as tf; \
print("Num GPUs Available: ",len(tf.config.experimental.list_physical_devices("GPU")))'
</pre>

'''Note:''' Ready-to-use containers can be pulled from the [https://ngc.nvidia.com/catalog/containers NVIDIA GPU CLOUD (NGC)] catalog.

== NGC environment modules on JUSTUS 2 ==

1) Prepare a workspace to store the container
<pre>
$ srun --nodes=1 --exclusive --gres=gpu:2 --pty bash
$ WORKSPACE=`ws_allocate ngc 3`
$ cd $WORKSPACE
$ export NGC_IMAGE_DIR=$(pwd)
</pre>
Important: Containers can only run in workspaces.

2) PyTorch container
<pre>
$ module load ngc/.numlib
$ module load 20.12-torch-py3
$ python3
>>> import torch
>>> x = torch.randn(2,3)
>>> print(x)
>>> quit()
$ module unload 20.12-torch-py3
$ module unload ngc/.numlib
</pre>
'''Note:''' Use the container in the same manner as an interactive shell.

3) LAMMPS container
<pre>
$ module load ngc/.chem
$ module load lammps-29Oct2020
$ wget https://lammps.sandia.gov/inputs/in.lj.txt
$ export SINGULARITY_BINDPATH=$(pwd)
$ mpirun -n 2 lmp -in in.lj.txt -var x 8 -var y 8 -var z 8 -k on g 2 -sf kk -pk kokkos cuda/aware on neigh full \
comm device binsize 2.8
$ module unload lammps-29Oct2020
$ module unload ngc/.chem
</pre>
'''Note:''' Use SINGULARITY_BINDPATH=<PATH> to mount the directory with the input file.

Currently, module load ngc/.numlib for numeric libraries and module load ngc/.chem for chemistry programs can be selected.

== Batch jobs with containers on JUSTUS 2 ==

Run a GROMACS container with Singularity as a batch job:
<pre>
$ WORKSPACE=`ws_allocate gromacs 3` # allocate workspace
$ cd $WORKSPACE # change to workspace
$ singularity pull gromacs-2020_2.sif docker://nvcr.io/hpc/gromacs:2020.2 # pull container from NGC
$ cp -r /opt/bwhpc/common/chem/ngc/gromacs/ ./bwhpc-examples/ # copy example to workspace
$ cd ./bwhpc-examples # change to example directory
$ sbatch gromacs-2020.2_gpu.slurm # submit job
$ squeue # obtain JOBID
$ scontrol show job <JOBID> # check state of job
</pre>

More batch job examples are located at /opt/bwhpc/common/chem/ngc.

= Useful links =

* [https://sylabs.io/docs/ Documentation (english)]
* [https://de.wikipedia.org/wiki/Singularity_(Software) Wikipedia article (german)]
* [https://en.wikipedia.org/wiki/Singularity_(software) Wikipedia article (english)]
----
[[Category:Scientific_Applications]][[Category:BwForCluster_Chemistry]][[Category:BwForCluster_JUSTUS_2]]

BwUniCluster3.0/Software/OpenFoam

2021-06-07T14:32:57Z

F Wagner:

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/openfoam
|-
| Availability
| [[bwUniCluster_2.0]] | [[BwForCluster_JUSTUS_2]] | bwGRiD-Tübingen | [[BwForCluster_MLS&WISO_Production]]
|-
| License
| [https://www.openfoam.org/licence.php GNU General Public Licence]
|-
| Citing
| n/a
|-
| Links
| [https://www.openfoam.org/ Homepage] | [https://www.openfoam.org/docs/ Documentation]
|-
| Graphical Interface
| No
|}
 
 
= Description =

'''OpenFOAM''' (Open-source Field Operation And Manipulation) is a free, open-source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.

= Availability =

OpenFOAM is available on selected bwHPC-Clusters. A complete list of versions currently installed on the bwHPC-Clusters can be obtained from the [https://www.bwhpc.de/software.html Cluster Information System (CIS)].

= Adding OpenFOAM to Your Environment =

In order to check which versions of OpenFOAM are installed on the compute cluster, run the following command:
<pre>$ module avail cae/openfoam</pre>

In general, several OpenFOAM versions should be returned.
The default version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam</pre>

Other OpenFOAM versions (if available) can be loaded according to:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, type
<pre>$ source $FOAM_INIT</pre>
or simply
<pre>$ foamInit</pre>

= Parallel run with OpenFOAM =
For a better performance on running OpenFOAM jobs in parallel on bwUniCluster, it is recommended to have the decomposed data in local folders on each node.

Therefore you may use *HPC scripts, wich will copy your data to the node specific folders after running the decomposePar, and copy it back to the local case folder before running reconstructPar.

Don't forget to allocate enough wall-time for decomposition and reconstruction of your cases. As the data will be processed directly on the nodes, and may be lost if the job is cancelled before the data is copied back into the case folder.

Following commands will do that for you:

<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>

instead of:
<pre>$ decomposePar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMesh -constant</pre>
 

For running jobs on multiple nodes, OpenFOAM needs passwordless communication between the nodes, to copy data into the local folders.

A small trick using ssh-keygen once will let your nodes to communicate freely over rsh.

Do it once (if you didn't do it already in the past):

<pre>
$ ssh-keygen
$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
</pre>

= Building an OpenFOAM batch file for parallel processing =
== General information ==
Before running OpenFOAM jobs in parallel, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use.

That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'', letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between.

There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results.

Decomposition and segments building process is handled bydecomposeParutility.

The number of subdomains, in which the geometry will be decomposed, is specified in "''system/decomposeParDict''", as well as the decomposition method to use.

The automatic decomposition method is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with not enough cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods.
 

== Wrapper script generation ==
'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 8, on 80 processors, on a ''multiple'' partition with a total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
# Allocate nodes
#SBATCH --nodes=2
# Number of tasks per node
#SBATCH --ntasks-per-node=40
# Queue class https://wiki.bwhpc.de/e/BwUniCluster_2.0_Batch_Queues
#SBATCH --partition=multiple
# Maximum job run time
#SBATCH --time=4:00:00
# Give the job a reasonable name
#SBATCH --job-name=openfoam
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=logs-%j.out
# File name for error output
#SBATCH --error=logs-%j.err

# User defined variables
FOAM_VERSION="8"
EXECUTABLE="icoFoam"
MPIRUN_OPTIONS="--bind-to core --map-by core --report-bindings"

module load ${FOAM_VERSION}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposeParHPC &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable
mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel &&

reconstructParHPC

</source>
|}
 
'''Attention:''' The script above will run a parallel OpenFOAM Job with pre-installed OpenMPI. If you are using an OpenFOAM version wich comes with pre-installed Intel MPI (like, for examplecae/openfoam/v1712-impi) you will have to modify the batch script to use all the advantages of Intel MPI for parallel calculations. For details see:
* [[Batch_Jobs_-_bwUniCluster_Features|Batch Jobs Features]]

= Using I/O and reducing the amount of data and files =
In OpenFOAM, you can control which variables or fields are written at specific times. For example, for post-processing purposes, you might need only a subset of variables. In order to control which files will be written, there is a function object called "writeObjects".

An example controlDict file may look like this: At the top of the file (entry "writeControl") you specify that ALL fields (variables) required for restarting are saved every 12 wall-clock hours. Then, additionally, at the bottom of the controlDict in the "functions" block, you can add a function object of type "writeObjects". With this function object, you can control the output of specific fields independent of the entry at the top of the file:

{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="text">
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.1.x |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

startFrom latestTime;
startTime 0;
stopAt endTime;
endTime 1e2;
deltaT 1e-5;

writeControl clockTime;
writeInterval 43200; // write ALL fields necessary to restart your simulation
// every 43200 wall-clock seconds = 12 hours of real time

purgeWrite 0;
writeFormat binary;
writePrecision 10;
writeCompression off;
timeFormat general;
timePrecision 10;
runTimeModifiable false;

functions
{
writeFields // name of the function object
{
type writeObjects;
libs ( "libutilityFunctionObjects.so" );

objects
(
T U rho // list of fields/variables to be written
);

// E.g. write every 1e-5 seconds of simulation time only the specified fields
writeControl runTime;
writeInterval 1e-5; // write every 1e-5 seconds
}
}

</source>
|}
 

You can also define multiple function objects in order to write different subsets of fields at different times. You can also use wildcards in the list of fields- for example, in order to write out all fields starting with "RR_" you can add
<pre>
"RR_.*"
</pre>
to the list of objects. You can get a list of valid field names by writing "banana" in the field list. During the run of the solver all valid field names are printed.
The output time can be changed too. Instead of writing at specific times in the simulation, you can also write after a certain number of time steps or depening on the wall clock time:

<pre>// write every 100th simulation time step
writeControl timeStep;
writeInterval 100;
</pre>

<pre>// every 3600 seconds of real wall clock time
writeControl runtime;
writeInterval 3600;
</pre>

If you use OpenFOAM before version 4.0 or 1606, the type of function object is:
<pre>
type writeRegisteredObject; // (instead of type writeObjects)
</pre>
If you use OpenFOAM before version 3.0, you have to load the library with
<pre>
functionObjectLibs ("libIOFunctionObjects.so"); // (instead of libs ( "libutilityFunctionObjects.so" ))
</pre>
and exchange the entry "writeControl" with "outputControl".

= OpenFOAM and ParaView on bwUniCluster=
ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/4.3.1</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
<pre>$ paraview</pre>
'''NOTICE:''' ParaView is a visualization software which requires X-server to run.

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.

----
[[Category:Engineering software]]
[[Category:BwUniCluster]]
[[Category:BwUniCluster_2.0]]
[[Category:BwForCluster_Chemistry]]
[[Category:BwForCluster_JUSTUS_2]]
[[Category:BwForCluster_MLS&WISO_Production]]

BwUniCluster3.0/Software/OpenFoam

2021-06-07T14:32:16Z

F Wagner: /* OpenFOAM and ParaView on bwUniCluster */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/openfoam
|-
| Availability
| [[bwUniCluster_2.0]] | [[BwForCluster_JUSTUS_2]] | bwGRiD-Tübingen
|-
| License
| [https://www.openfoam.org/licence.php GNU General Public Licence]
|-
| Citing
| n/a
|-
| Links
| [https://www.openfoam.org/ Homepage] | [https://www.openfoam.org/docs/ Documentation]
|-
| Graphical Interface
| No
|}
 
 
= Description =

'''OpenFOAM''' (Open-source Field Operation And Manipulation) is a free, open-source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.

= Availability =

OpenFOAM is available on selected bwHPC-Clusters. A complete list of versions currently installed on the bwHPC-Clusters can be obtained from the [https://www.bwhpc.de/software.html Cluster Information System (CIS)].

= Adding OpenFOAM to Your Environment =

In order to check which versions of OpenFOAM are installed on the compute cluster, run the following command:
<pre>$ module avail cae/openfoam</pre>

In general, several OpenFOAM versions should be returned.
The default version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam</pre>

Other OpenFOAM versions (if available) can be loaded according to:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, type
<pre>$ source $FOAM_INIT</pre>
or simply
<pre>$ foamInit</pre>

= Parallel run with OpenFOAM =
For a better performance on running OpenFOAM jobs in parallel on bwUniCluster, it is recommended to have the decomposed data in local folders on each node.

Therefore you may use *HPC scripts, wich will copy your data to the node specific folders after running the decomposePar, and copy it back to the local case folder before running reconstructPar.

Don't forget to allocate enough wall-time for decomposition and reconstruction of your cases. As the data will be processed directly on the nodes, and may be lost if the job is cancelled before the data is copied back into the case folder.

Following commands will do that for you:

<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>

instead of:
<pre>$ decomposePar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMesh -constant</pre>
 

For running jobs on multiple nodes, OpenFOAM needs passwordless communication between the nodes, to copy data into the local folders.

A small trick using ssh-keygen once will let your nodes to communicate freely over rsh.

Do it once (if you didn't do it already in the past):

<pre>
$ ssh-keygen
$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
</pre>

= Building an OpenFOAM batch file for parallel processing =
== General information ==
Before running OpenFOAM jobs in parallel, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use.

That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'', letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between.

There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results.

Decomposition and segments building process is handled bydecomposeParutility.

The number of subdomains, in which the geometry will be decomposed, is specified in "''system/decomposeParDict''", as well as the decomposition method to use.

The automatic decomposition method is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with not enough cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods.
 

== Wrapper script generation ==
'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 8, on 80 processors, on a ''multiple'' partition with a total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
# Allocate nodes
#SBATCH --nodes=2
# Number of tasks per node
#SBATCH --ntasks-per-node=40
# Queue class https://wiki.bwhpc.de/e/BwUniCluster_2.0_Batch_Queues
#SBATCH --partition=multiple
# Maximum job run time
#SBATCH --time=4:00:00
# Give the job a reasonable name
#SBATCH --job-name=openfoam
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=logs-%j.out
# File name for error output
#SBATCH --error=logs-%j.err

# User defined variables
FOAM_VERSION="8"
EXECUTABLE="icoFoam"
MPIRUN_OPTIONS="--bind-to core --map-by core --report-bindings"

module load ${FOAM_VERSION}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposeParHPC &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable
mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel &&

reconstructParHPC

</source>
|}
 
'''Attention:''' The script above will run a parallel OpenFOAM Job with pre-installed OpenMPI. If you are using an OpenFOAM version wich comes with pre-installed Intel MPI (like, for examplecae/openfoam/v1712-impi) you will have to modify the batch script to use all the advantages of Intel MPI for parallel calculations. For details see:
* [[Batch_Jobs_-_bwUniCluster_Features|Batch Jobs Features]]

= Using I/O and reducing the amount of data and files =
In OpenFOAM, you can control which variables or fields are written at specific times. For example, for post-processing purposes, you might need only a subset of variables. In order to control which files will be written, there is a function object called "writeObjects".

An example controlDict file may look like this: At the top of the file (entry "writeControl") you specify that ALL fields (variables) required for restarting are saved every 12 wall-clock hours. Then, additionally, at the bottom of the controlDict in the "functions" block, you can add a function object of type "writeObjects". With this function object, you can control the output of specific fields independent of the entry at the top of the file:

{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="text">
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.1.x |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

startFrom latestTime;
startTime 0;
stopAt endTime;
endTime 1e2;
deltaT 1e-5;

writeControl clockTime;
writeInterval 43200; // write ALL fields necessary to restart your simulation
// every 43200 wall-clock seconds = 12 hours of real time

purgeWrite 0;
writeFormat binary;
writePrecision 10;
writeCompression off;
timeFormat general;
timePrecision 10;
runTimeModifiable false;

functions
{
writeFields // name of the function object
{
type writeObjects;
libs ( "libutilityFunctionObjects.so" );

objects
(
T U rho // list of fields/variables to be written
);

// E.g. write every 1e-5 seconds of simulation time only the specified fields
writeControl runTime;
writeInterval 1e-5; // write every 1e-5 seconds
}
}

</source>
|}
 

You can also define multiple function objects in order to write different subsets of fields at different times. You can also use wildcards in the list of fields- for example, in order to write out all fields starting with "RR_" you can add
<pre>
"RR_.*"
</pre>
to the list of objects. You can get a list of valid field names by writing "banana" in the field list. During the run of the solver all valid field names are printed.
The output time can be changed too. Instead of writing at specific times in the simulation, you can also write after a certain number of time steps or depening on the wall clock time:

<pre>// write every 100th simulation time step
writeControl timeStep;
writeInterval 100;
</pre>

<pre>// every 3600 seconds of real wall clock time
writeControl runtime;
writeInterval 3600;
</pre>

If you use OpenFOAM before version 4.0 or 1606, the type of function object is:
<pre>
type writeRegisteredObject; // (instead of type writeObjects)
</pre>
If you use OpenFOAM before version 3.0, you have to load the library with
<pre>
functionObjectLibs ("libIOFunctionObjects.so"); // (instead of libs ( "libutilityFunctionObjects.so" ))
</pre>
and exchange the entry "writeControl" with "outputControl".

= OpenFOAM and ParaView on bwUniCluster=
ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/4.3.1</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
<pre>$ paraview</pre>
'''NOTICE:''' ParaView is a visualization software which requires X-server to run.

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.

----
[[Category:Engineering software]]
[[Category:BwUniCluster]]
[[Category:BwUniCluster_2.0]]
[[Category:BwForCluster_Chemistry]]
[[Category:BwForCluster_JUSTUS_2]]
[[Category:BwForCluster_MLS&WISO_Production]]

JUSTUS2/Software/NAMD

2021-06-07T14:30:43Z

F Wagner: /* Useful links */

{| width=760px class="wikitable"
|-
! Description !! Content
|-
| module load
| chem/namd
|-
| Availability
| [[BwForCluster_JUSTUS_2]] | [[BwForCluster_MLS&WISO_Production]]
|-
| License
| Freeware for non-commercial use. [[#License|More...]]
|-
| Citing
| [http://www.ks.uiuc.edu/Research/namd/papers.html Publications]
|-
| Links
| [http://www.ks.uiuc.edu/Research/namd/ Homepage] | [http://www.ks.uiuc.edu/Research/namd/2.14/ug/ Documentation]
|-
| Graphical Interface
| No
|}

= Description =

'''NAMD''' (NAnoscale Molecular Dynamics) is a parallel molecular dynamics code designed for high-performance simulation of large biomolecular systems.

= Availability =

NAMD is available on selected bwHPC-Clusters. A complete list of versions currently installed on the bwHPC-Clusters can be obtained from the [https://www.bwhpc.de/software.html Cluster Information System (CIS)].

In order to check which versions of NAMD are installed on the compute cluster, run the following command:
<pre>
$ module avail chem/namd
</pre>

= License =

NAMD is available as [https://en.wikipedia.org/wiki/Freeware freeware] for non-commercial use by individuals, academic institutions, and corporations for in-house business uses. Please read the [http://www.ks.uiuc.edu/Research/namd/license.html license] for additional information about NAMD.

= Usage =

== Loading the module ==

You can load the default version of NAMD with the following command:
<pre>
$ module load chem/namd
</pre>

The module will try to load all modules (i.e., compiler, MPI, MKL) it needs to function. If loading the module fails, check if you have already loaded one of those modules, but not in the version required by NAMD.

If you wish to load another (older) version of NAMD, you can do so using
<pre>
$ module load chem/namd/<version>
</pre>
with <version> specifying the desired version.

Please cite NAMD in your publications according to the [http://www.ks.uiuc.edu/Research/namd/papers.html references].

== Program Binaries ==

The binary ''namd2'' is main program of the NAMD package.
<pre>
Usage: namd2 [options] [config file]

Example: namd2 +isomalloc_sync run.namd
</pre>

For information about how to construct NAMD configuration files, please see the [http://www.ks.uiuc.edu/Research/namd/2.14/ug/ documentation]. In addition to a configuration file, NAMD also needs a CHARMM force field in either CHARMM or X-PLOR format, an X-PLOR format PSF file describing the molecular structure, and the initial coordinates of the molecular system in the form of a PDB file.

The NAMD package includes several additional tools:

* flipbinpdb -- flips byte-ordering of 8-byte doubles
* flipdcd -- flips byte-ordering of DCD files
* psfgen -- [https://www.ks.uiuc.edu/Research/vmd/ VMD] psfgen plugin
* sortreplicas -- un-shuffles replica trajectories to place same-temperature frames in the same file

=== Charm++ ===

Charm++ is a parallel object-oriented programming paradigm based on C++ and developed in the Parallel Programming Laboratory at the University of Illinois at Urbana–Champaign. NAMD has been implemented using Charm++.

* charmrun -- launches Charm++ programs

For help on usage, type
<pre>
$ charmrun -help
</pre>

=== GPU Acceleration ===

Currently not available.

== Hints for using NAMD ==

=== Parallel Execution ===

It is usually more efficient to run several smaller jobs with reasonable speedup side by side. Check for a reasonable speedup by increasing the number of workers from 1, 2, 4, n cores. Do not forget to adjust the memory specification when changing the number of workers

=== Memory Management ===

It requires some experience to choose the perfect memory value. Monitoring a job interactively may help to estimate the memory consumption. Requesting 1GB more than the required amount is typically sufficient.

=== Runtime Specification ===

The wall-time limit is independent of the number of parallel workers. Selecting the best value requires some experience. In general, jobs with shorter wall times get higher priorities. It could help to run the job interactively for a small period of time to monitor its convergence. Furthermore, it may be advantageous to extrapolate the run time by looking at smaller test jobs.

= Examples =

As with all processes that require more than a few minutes to run, non-trivial compute jobs must be submitted to the cluster queuing system.

Example scripts are available in the directory $NAMD_EXA_DIR:
<pre>
$ module show chem/namd # show environment variables, which will be available after 'module load'
$ module load chem/namd # load module
$ ls $NAMD_EXA_DIR # show content of directory $NAMD_EXA_DIR
$ cat $NAMD_EXA_DIR/README # show examples README
</pre>

Run a first simple example job:
<pre>
$ module load chem/namd # load module
$ WORKSPACE=`ws_allocate namd 3` # allocate workspace
$ cd $WORKSPACE # change to workspace
$ cp -a $NAMD_HOME/bwhpc-examples . # copy example files to workspace
$ cd bwhpc-examples/apoa1 # change to test directory
$ sbatch bwhpc_namd_apoa1.sh # submit job
$ squeue # obtain JOBID
$ scontrol show job <JOBID> # check state of job
$ ls # when job finishes the results will be visible in this directory
</pre>

= Useful links =

* [http://www.ks.uiuc.edu/Research/namd/2.14/ug/ Documentation (english)]
* [http://www.ks.uiuc.edu/Research/namd/mailing_list/ Maling list (english)]
* [https://en.wikipedia.org/wiki/NAMD Wikipedia article (english)]
* [https://de.wikipedia.org/wiki/NAMD Wikipedia article (german)]
----
[[Category:Chemistry software]][[Category:BwForCluster_Chemistry]][[Category:BwForCluster_JUSTUS_2]][[Category:BwForCluster_MLS&WISO_Production]]

JUSTUS2/Software/NAMD

2021-06-02T22:32:12Z

F Wagner: /* Compile */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| numlib/gsl
|-
| Availability
| [[bwUniCluster_2.0]] | [[BwForCluster_JUSTUS_2]]
|-
| License
| [http://www.gnu.org/copyleft/gpl.html GNU General Public License (GPL)]
|-
|Citing
| <pre>M. Galassi et al., GNU Scientific Library Reference Manual (3rd Ed.), ISBN 0954612078.</pre>
|-
| Links
| [https://www.gnu.org/software/gsl/ Homepage] | [https://www.gnu.org/software/gsl/doc/html/index.html Documentation]
|-
| Graphical Interface
| No
|-
|}
 
= Description =
The '''GNU Scientific Library''' ('''GSL''') is a software library for numerical computations in applied mathematics and science. GSL is written in C, but bindings exist for other programming languages as well. The library provides a wide range of mathematical routines such as random number generators, special functions and least-squares fitting. There are over 1000 functions in total with an extensive test suite.
 
 
The complete range of subject areas covered by the library:
 
 
• Complex Numbers • Roots of Polynomials • Special Functions • Vectors and Matrices • Permutations • Sorting • BLAS Support • Linear Algebra • Eigensystems • Fast Fourier Transforms • Quadrature • Random Numbers • Quasi-Random Sequences • Random Distributions • Statistics • Histograms • N-Tuples • Monte Carlo Integration • Simulated Annealing • Differential Equations • Interpolation • Numerical Differentiation • Chebyshev Approximation • Series Acceleration • Discrete Hankel Transforms • Root-Finding • Minimization • Least-Squares Fitting • Physical Constants • IEEE Floating-Point • Discrete Wavelet Transforms • Basis splines • Running Statistics • Sparse Matrices and Linear Algebra
 
 

= Availability =

GSL is available on selected bwHPC-Clusters. A complete list of versions currently installed on the bwHPC-Clusters can be obtained from the [https://www.bwhpc.de/software.html Cluster Information System (CIS)].

In order to check which versions of GSL are installed on the compute cluster, run the following command:
<pre>
$ module avail numlib/gsl
</pre>

= Documentation =

A documentation for GSL is available [https://www.gnu.org/software/gsl/doc/html/index.html online].

The help page of the GSL module provides more version specific information:
<pre>
$ module help numlib/gsl

----------- Module Specific Help for 'numlib/gsl/2.6' ----------
This module sets the path and environment variables for GSL-2.6.

The library has been compiled with GNU compiler 8.3 using the following optimization flags:

-O3 -funroll-loops -march=native -mtune=native

Online-Documentation: $GSL_WWW

Local-Documentation: $GSL_DOC_DIR

Code Examples: $GSL_EXA_DIR

Tips for compiling and linking:

After having loaded the environment module, you can use several environment variables to
compile and link your application with the GSL library.

Your source code should contain preprocessor include statements with a gsl/prefix, such as

#include <gsl/gsl_math.h>

A typical compilation command for a source file example.c with the GNU C compiler gcc is

gcc -Wall -I\$GSL_INC_DIR -c example.c

The GSL_INC_DIR environment variable points to the location of the include path for the GSL header files.

The following command can be used to link the application with the GSL library:

gcc -L\$GSL_LIB_DIR -o example example.o -lgsl -lgslcblas -lm

The GSL_LIB_DIR environment variable points to the location of the GSL library.

In case of problems, submit a trouble ticket at 'https://bw-support.scc.kit.edu'.

The full version is: numlib/gsl/2.6
</pre>

= Usage =

== Loading the module ==

You can load the default version of GSL with the following command:
<pre>
$ module load numlib/gsl
</pre>

The module will try to load all modules it needs to function (e.g., compiler, mpi, ...). If loading the module fails, check if you have already loaded one of those modules, but not in the version required by GSL.

If you wish to load another (older) version of GSL, you can do so using
<pre>
$ module load numlib/gsl/<version>
</pre>
with <version> specifying the desired version.

== How to use GSL ==

A man page is available and can be accessed by typing:
<pre>
$ man gsl
</pre>

=== Includes ===

Your source code should contain preprocessor include statements with a gsl/prefix such as
<source lang="c">#include <gsl/gsl_math.h></source>

=== Compile ===

A typical compilation command for a source file 'example.c' with the Intel C compiler icc is
<pre> $ icc -Wall -I$GSL_INC_DIR -c example.c </pre>
The $GSL_INC_DIR environment variable points to location of the include path for the GSL header files.

=== Link ===

The following command can be used to link the application with the GSL libraries.
<pre> $ icc -L$GSL_LIB_DIR -o example example.o -lgsl -lgslcblas -lm </pre>
The [[#GSL-Specific Environments|$GSL_LIB_DIR environment variable]] points to the location
of the gsl libraries.
 
Also make sure to have the GSL-module loaded before running applications build
with this library.
 

= Examples =

Consider the following GSL task and create source code file 'gsl-test.c':

<source lang="c">
#include <stdio.h>
#include <gsl/gsl_sf_bessel.h>

int main (void)
{
double x = 5.0;
double y = gsl_sf_bessel_J0 (x);
printf ("J0(%g) = %.18e\n", x, y);
return 0;
}
</source>

The code can be compiled and executed with the following commands:

<pre>
$ module load numlib/gsl/2.6-intel-19.1.2
$ icc -Wall -I$GSL_INC_DIR -c gsl-test.c
$ icc -L$GSL_LIB_DIR -o example gsl-test.o -lgsl -lgslcblas -lm
$ ./example
J0(5) = -1.775967713143382642e-01
</pre>

= Useful links =

* [https://www.gnu.org/software/gsl/doc/html/index.html Documentation (english)]
* [https://de.wikipedia.org/wiki/GNU_Scientific_Library Wikipedia article (german)]
* [https://en.wikipedia.org/wiki/GNU_Scientific_Library Wikipedia article (english)]
----
[[Category: Numerical libraries]][[Category:BwUniCluster]][[Category:BwUniCluster_2.0]][[Category:BwForCluster_Chemistry]][[Category:BwForCluster_JUSTUS_2]]