bwHPC Wiki - User contributions [en]

BwUniCluster3.0/Software/OpenFoam

2021-07-29T09:22:21Z

A Saramet: /* OpenFOAM and ParaView on bwUniCluster */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/openfoam
|-
| Availability
| [[bwUniCluster_2.0]] | [[BwForCluster_JUSTUS_2]] | bwGRiD-Tübingen | [[BwForCluster_MLS&WISO_Production]]
|-
| License
| [https://www.openfoam.org/licence.php GNU General Public Licence]
|-
| Citing
| n/a
|-
| Links
| [https://www.openfoam.org/ Homepage] | [https://www.openfoam.org/docs/ Documentation]
|-
| Graphical Interface
| No
|}
 
 
= Description =

'''OpenFOAM''' (Open-source Field Operation And Manipulation) is a free, open-source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.

= Availability =

OpenFOAM is available on selected bwHPC-Clusters. A complete list of versions currently installed on the bwHPC-Clusters can be obtained from the [https://www.bwhpc.de/software.html Cluster Information System (CIS)].

= Adding OpenFOAM to Your Environment =

In order to check which versions of OpenFOAM are installed on the compute cluster, run the following command:
<pre>$ module avail cae/openfoam</pre>

In general, several OpenFOAM versions should be returned.
The default version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam</pre>

Other OpenFOAM versions (if available) can be loaded according to:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, type
<pre>$ source $FOAM_INIT</pre>
or simply
<pre>$ foamInit</pre>

= Parallel run with OpenFOAM =
For a better performance on running OpenFOAM jobs in parallel on bwUniCluster, it is recommended to have the decomposed data in local folders on each node.

Therefore you may use *HPC scripts, wich will copy your data to the node specific folders after running the decomposePar, and copy it back to the local case folder before running reconstructPar.

Don't forget to allocate enough wall-time for decomposition and reconstruction of your cases. As the data will be processed directly on the nodes, and may be lost if the job is cancelled before the data is copied back into the case folder.

Following commands will do that for you:

<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>

instead of:
<pre>$ decomposePar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMesh -constant</pre>
 

For running jobs on multiple nodes, OpenFOAM needs passwordless communication between the nodes, to copy data into the local folders.

A small trick using ssh-keygen once will let your nodes to communicate freely over rsh.

Do it once (if you didn't do it already in the past):

<pre>
$ ssh-keygen
$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
</pre>

= Building an OpenFOAM batch file for parallel processing =
== General information ==
Before running OpenFOAM jobs in parallel, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use.

That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'', letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between.

There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results.

Decomposition and segments building process is handled bydecomposeParutility.

The number of subdomains, in which the geometry will be decomposed, is specified in "''system/decomposeParDict''", as well as the decomposition method to use.

The automatic decomposition method is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with not enough cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods.
 

== Wrapper script generation ==
'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 8, on 80 processors, on a ''multiple'' partition with a total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
# Allocate nodes
#SBATCH --nodes=2
# Number of tasks per node
#SBATCH --ntasks-per-node=40
# Queue class https://wiki.bwhpc.de/e/BwUniCluster_2.0_Batch_Queues
#SBATCH --partition=multiple
# Maximum job run time
#SBATCH --time=4:00:00
# Give the job a reasonable name
#SBATCH --job-name=openfoam
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=logs-%j.out
# File name for error output
#SBATCH --error=logs-%j.err

# User defined variables
FOAM_VERSION="8"
EXECUTABLE="icoFoam"
MPIRUN_OPTIONS="--bind-to core --map-by core --report-bindings"

module load ${FOAM_VERSION}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposeParHPC &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable
mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel &&

reconstructParHPC

</source>
|}
 
'''Attention:''' The script above will run a parallel OpenFOAM Job with pre-installed OpenMPI. If you are using an OpenFOAM version wich comes with pre-installed Intel MPI (like, for examplecae/openfoam/v1712-impi) you will have to modify the batch script to use all the advantages of Intel MPI for parallel calculations. For details see:
* [[Batch_Jobs_-_bwUniCluster_Features|Batch Jobs Features]]

= Using I/O and reducing the amount of data and files =
In OpenFOAM, you can control which variables or fields are written at specific times. For example, for post-processing purposes, you might need only a subset of variables. In order to control which files will be written, there is a function object called "writeObjects".

An example controlDict file may look like this: At the top of the file (entry "writeControl") you specify that ALL fields (variables) required for restarting are saved every 12 wall-clock hours. Then, additionally, at the bottom of the controlDict in the "functions" block, you can add a function object of type "writeObjects". With this function object, you can control the output of specific fields independent of the entry at the top of the file:

{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="text">
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.1.x |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

startFrom latestTime;
startTime 0;
stopAt endTime;
endTime 1e2;
deltaT 1e-5;

writeControl clockTime;
writeInterval 43200; // write ALL fields necessary to restart your simulation
// every 43200 wall-clock seconds = 12 hours of real time

purgeWrite 0;
writeFormat binary;
writePrecision 10;
writeCompression off;
timeFormat general;
timePrecision 10;
runTimeModifiable false;

functions
{
writeFields // name of the function object
{
type writeObjects;
libs ( "libutilityFunctionObjects.so" );

objects
(
T U rho // list of fields/variables to be written
);

// E.g. write every 1e-5 seconds of simulation time only the specified fields
writeControl runTime;
writeInterval 1e-5; // write every 1e-5 seconds
}
}

</source>
|}
 

You can also define multiple function objects in order to write different subsets of fields at different times. You can also use wildcards in the list of fields- for example, in order to write out all fields starting with "RR_" you can add
<pre>
"RR_.*"
</pre>
to the list of objects. You can get a list of valid field names by writing "banana" in the field list. During the run of the solver all valid field names are printed.
The output time can be changed too. Instead of writing at specific times in the simulation, you can also write after a certain number of time steps or depening on the wall clock time:

<pre>// write every 100th simulation time step
writeControl timeStep;
writeInterval 100;
</pre>

<pre>// every 3600 seconds of real wall clock time
writeControl runtime;
writeInterval 3600;
</pre>

If you use OpenFOAM before version 4.0 or 1606, the type of function object is:
<pre>
type writeRegisteredObject; // (instead of type writeObjects)
</pre>
If you use OpenFOAM before version 3.0, you have to load the library with
<pre>
functionObjectLibs ("libIOFunctionObjects.so"); // (instead of libs ( "libutilityFunctionObjects.so" ))
</pre>
and exchange the entry "writeControl" with "outputControl".

= OpenFOAM and ParaView on bwUniCluster=
ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/5.9</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
To run Paraview using VNC system is required on the bwUniCluster.
On the cluster run:
<pre>$ start_vnc_desktop --hw-rendering </pre>
Start your VNC client on your desktop PC.
'''NOTICE''' Information for remote visualization on KIT HPC system is available on: https://wiki.scc.kit.edu/hpc/index.php/Remote_Visualization

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.

----
[[Category:Engineering software]]
[[Category:BwUniCluster]]
[[Category:BwUniCluster_2.0]]
[[Category:BwForCluster_Chemistry]]
[[Category:BwForCluster_JUSTUS_2]]
[[Category:BwForCluster_MLS&WISO_Production]]

BwUniCluster3.0/Software/OpenFoam

2021-07-29T09:19:42Z

A Saramet: /* OpenFOAM and ParaView on bwUniCluster */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/openfoam
|-
| Availability
| [[bwUniCluster_2.0]] | [[BwForCluster_JUSTUS_2]] | bwGRiD-Tübingen | [[BwForCluster_MLS&WISO_Production]]
|-
| License
| [https://www.openfoam.org/licence.php GNU General Public Licence]
|-
| Citing
| n/a
|-
| Links
| [https://www.openfoam.org/ Homepage] | [https://www.openfoam.org/docs/ Documentation]
|-
| Graphical Interface
| No
|}
 
 
= Description =

'''OpenFOAM''' (Open-source Field Operation And Manipulation) is a free, open-source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.

= Availability =

OpenFOAM is available on selected bwHPC-Clusters. A complete list of versions currently installed on the bwHPC-Clusters can be obtained from the [https://www.bwhpc.de/software.html Cluster Information System (CIS)].

= Adding OpenFOAM to Your Environment =

In order to check which versions of OpenFOAM are installed on the compute cluster, run the following command:
<pre>$ module avail cae/openfoam</pre>

In general, several OpenFOAM versions should be returned.
The default version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam</pre>

Other OpenFOAM versions (if available) can be loaded according to:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, type
<pre>$ source $FOAM_INIT</pre>
or simply
<pre>$ foamInit</pre>

= Parallel run with OpenFOAM =
For a better performance on running OpenFOAM jobs in parallel on bwUniCluster, it is recommended to have the decomposed data in local folders on each node.

Therefore you may use *HPC scripts, wich will copy your data to the node specific folders after running the decomposePar, and copy it back to the local case folder before running reconstructPar.

Don't forget to allocate enough wall-time for decomposition and reconstruction of your cases. As the data will be processed directly on the nodes, and may be lost if the job is cancelled before the data is copied back into the case folder.

Following commands will do that for you:

<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>

instead of:
<pre>$ decomposePar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMesh -constant</pre>
 

For running jobs on multiple nodes, OpenFOAM needs passwordless communication between the nodes, to copy data into the local folders.

A small trick using ssh-keygen once will let your nodes to communicate freely over rsh.

Do it once (if you didn't do it already in the past):

<pre>
$ ssh-keygen
$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
</pre>

= Building an OpenFOAM batch file for parallel processing =
== General information ==
Before running OpenFOAM jobs in parallel, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use.

That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'', letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between.

There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results.

Decomposition and segments building process is handled bydecomposeParutility.

The number of subdomains, in which the geometry will be decomposed, is specified in "''system/decomposeParDict''", as well as the decomposition method to use.

The automatic decomposition method is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with not enough cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods.
 

== Wrapper script generation ==
'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 8, on 80 processors, on a ''multiple'' partition with a total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
# Allocate nodes
#SBATCH --nodes=2
# Number of tasks per node
#SBATCH --ntasks-per-node=40
# Queue class https://wiki.bwhpc.de/e/BwUniCluster_2.0_Batch_Queues
#SBATCH --partition=multiple
# Maximum job run time
#SBATCH --time=4:00:00
# Give the job a reasonable name
#SBATCH --job-name=openfoam
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=logs-%j.out
# File name for error output
#SBATCH --error=logs-%j.err

# User defined variables
FOAM_VERSION="8"
EXECUTABLE="icoFoam"
MPIRUN_OPTIONS="--bind-to core --map-by core --report-bindings"

module load ${FOAM_VERSION}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposeParHPC &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable
mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel &&

reconstructParHPC

</source>
|}
 
'''Attention:''' The script above will run a parallel OpenFOAM Job with pre-installed OpenMPI. If you are using an OpenFOAM version wich comes with pre-installed Intel MPI (like, for examplecae/openfoam/v1712-impi) you will have to modify the batch script to use all the advantages of Intel MPI for parallel calculations. For details see:
* [[Batch_Jobs_-_bwUniCluster_Features|Batch Jobs Features]]

= Using I/O and reducing the amount of data and files =
In OpenFOAM, you can control which variables or fields are written at specific times. For example, for post-processing purposes, you might need only a subset of variables. In order to control which files will be written, there is a function object called "writeObjects".

An example controlDict file may look like this: At the top of the file (entry "writeControl") you specify that ALL fields (variables) required for restarting are saved every 12 wall-clock hours. Then, additionally, at the bottom of the controlDict in the "functions" block, you can add a function object of type "writeObjects". With this function object, you can control the output of specific fields independent of the entry at the top of the file:

{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="text">
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.1.x |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

startFrom latestTime;
startTime 0;
stopAt endTime;
endTime 1e2;
deltaT 1e-5;

writeControl clockTime;
writeInterval 43200; // write ALL fields necessary to restart your simulation
// every 43200 wall-clock seconds = 12 hours of real time

purgeWrite 0;
writeFormat binary;
writePrecision 10;
writeCompression off;
timeFormat general;
timePrecision 10;
runTimeModifiable false;

functions
{
writeFields // name of the function object
{
type writeObjects;
libs ( "libutilityFunctionObjects.so" );

objects
(
T U rho // list of fields/variables to be written
);

// E.g. write every 1e-5 seconds of simulation time only the specified fields
writeControl runTime;
writeInterval 1e-5; // write every 1e-5 seconds
}
}

</source>
|}
 

You can also define multiple function objects in order to write different subsets of fields at different times. You can also use wildcards in the list of fields- for example, in order to write out all fields starting with "RR_" you can add
<pre>
"RR_.*"
</pre>
to the list of objects. You can get a list of valid field names by writing "banana" in the field list. During the run of the solver all valid field names are printed.
The output time can be changed too. Instead of writing at specific times in the simulation, you can also write after a certain number of time steps or depening on the wall clock time:

<pre>// write every 100th simulation time step
writeControl timeStep;
writeInterval 100;
</pre>

<pre>// every 3600 seconds of real wall clock time
writeControl runtime;
writeInterval 3600;
</pre>

If you use OpenFOAM before version 4.0 or 1606, the type of function object is:
<pre>
type writeRegisteredObject; // (instead of type writeObjects)
</pre>
If you use OpenFOAM before version 3.0, you have to load the library with
<pre>
functionObjectLibs ("libIOFunctionObjects.so"); // (instead of libs ( "libutilityFunctionObjects.so" ))
</pre>
and exchange the entry "writeControl" with "outputControl".

= OpenFOAM and ParaView on bwUniCluster=
ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/5.9</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
To run Paraview using VNC system is required on the bwUniCluster.
Run 'start_vnc_desktop --hw-rendering' and start your VNC client on your desktop PC.
Information for remote visualization on KIT HPC system is available on:
https://wiki.scc.kit.edu/hpc/index.php/Remote_Visualization

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.

----
[[Category:Engineering software]]
[[Category:BwUniCluster]]
[[Category:BwUniCluster_2.0]]
[[Category:BwForCluster_Chemistry]]
[[Category:BwForCluster_JUSTUS_2]]
[[Category:BwForCluster_MLS&WISO_Production]]

BwUniCluster3.0/Software/OpenFoam

2021-05-05T14:44:42Z

A Saramet: /* Wrapper script generation */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/openfoam
|-
| Availability
| [[bwUniCluster_2.0]] | [[BwForCluster_JUSTUS_2]] | bwGRiD-Tübingen
|-
| License
| [https://www.openfoam.org/licence.php GNU General Public Licence]
|-
| Citing
| n/a
|-
| Links
| [https://www.openfoam.org/ Homepage] | [https://www.openfoam.org/docs/ Documentation]
|-
| Graphical Interface
| No
|}
 
 
= Description =

'''OpenFOAM''' (Open-source Field Operation And Manipulation) is a free, open-source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.

= Availability =

OpenFOAM is available on selected bwHPC-Clusters. A complete list of versions currently installed on the bwHPC-Clusters can be obtained from the [https://www.bwhpc.de/software.html Cluster Information System (CIS)].

= Adding OpenFOAM to Your Environment =

In order to check which versions of OpenFOAM are installed on the compute cluster, run the following command:
<pre>$ module avail cae/openfoam</pre>

In general, several OpenFOAM versions should be returned.
The default version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam</pre>

Other OpenFOAM versions (if available) can be loaded according to:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, type
<pre>$ source $FOAM_INIT</pre>
or simply
<pre>$ foamInit</pre>

= Parallel run with OpenFOAM =
For a better performance on running OpenFOAM jobs in parallel on bwUniCluster, it is recommended to have the decomposed data in local folders on each node.

Therefore you may use *HPC scripts, wich will copy your data to the node specific folders after running the decomposePar, and copy it back to the local case folder before running reconstructPar.

Don't forget to allocate enough wall-time for decomposition and reconstruction of your cases. As the data will be processed directly on the nodes, and may be lost if the job is cancelled before the data is copied back into the case folder.

Following commands will do that for you:

<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>

instead of:
<pre>$ decomposePar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMesh -constant</pre>
 

For running jobs on multiple nodes, OpenFOAM needs passwordless communication between the nodes, to copy data into the local folders.

A small trick using ssh-keygen once will let your nodes to communicate freely over rsh.

Do it once (if you didn't do it already in the past):

<pre>
$ ssh-keygen
$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
</pre>

= Building an OpenFOAM batch file for parallel processing =
== General information ==
Before running OpenFOAM jobs in parallel, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use.

That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'', letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between.

There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results.

Decomposition and segments building process is handled bydecomposeParutility.

The number of subdomains, in which the geometry will be decomposed, is specified in "''system/decomposeParDict''", as well as the decomposition method to use.

The automatic decomposition method is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with not enough cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods.
 

== Wrapper script generation ==
'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 8, on 80 processors, on a ''multiple'' partition with a total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
# Allocate nodes
#SBATCH --nodes=2
# Number of tasks per node
#SBATCH --ntasks-per-node=40
# Queue class https://wiki.bwhpc.de/e/BwUniCluster_2.0_Batch_Queues
#SBATCH --partition=multiple
# Maximum job run time
#SBATCH --time=4:00:00
# Give the job a reasonable name
#SBATCH --job-name=openfoam
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=logs-%j.out
# File name for error output
#SBATCH --error=logs-%j.err

# User defined variables
FOAM_VERSION="8"
EXECUTABLE="icoFoam"
MPIRUN_OPTIONS="--bind-to core --map-by core --report-bindings"

module load ${FOAM_VERSION}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposeParHPC &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable
mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel &&

reconstructParHPC

</source>
|}
 
'''Attention:''' The script above will run a parallel OpenFOAM Job with pre-installed OpenMPI. If you are using an OpenFOAM version wich comes with pre-installed Intel MPI (like, for examplecae/openfoam/v1712-impi) you will have to modify the batch script to use all the advantages of Intel MPI for parallel calculations. For details see:
* [[Batch_Jobs_-_bwUniCluster_Features|Batch Jobs Features]]

= Using I/O and reducing the amount of data and files =
In OpenFOAM, you can control which variables or fields are written at specific times. For example, for post-processing purposes, you might need only a subset of variables. In order to control which files will be written, there is a function object called "writeObjects".

An example controlDict file may look like this: At the top of the file (entry "writeControl") you specify that ALL fields (variables) required for restarting are saved every 12 wall-clock hours. Then, additionally, at the bottom of the controlDict in the "functions" block, you can add a function object of type "writeObjects". With this function object, you can control the output of specific fields independent of the entry at the top of the file:

{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="text">
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.1.x |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

startFrom latestTime;
startTime 0;
stopAt endTime;
endTime 1e2;
deltaT 1e-5;

writeControl clockTime;
writeInterval 43200; // write ALL fields necessary to restart your simulation
// every 43200 wall-clock seconds = 12 hours of real time

purgeWrite 0;
writeFormat binary;
writePrecision 10;
writeCompression off;
timeFormat general;
timePrecision 10;
runTimeModifiable false;

functions
{
writeFields // name of the function object
{
type writeObjects;
libs ( "libutilityFunctionObjects.so" );

objects
(
T U rho // list of fields/variables to be written
);

// E.g. write every 1e-5 seconds of simulation time only the specified fields
writeControl runTime;
writeInterval 1e-5; // write every 1e-5 seconds
}
}

</source>
|}
 

You can also define multiple function objects in order to write different subsets of fields at different times. You can also use wildcards in the list of fields- for example, in order to write out all fields starting with "RR_" you can add
<pre>
"RR_.*"
</pre>
to the list of objects. You can get a list of valid field names by writing "banana" in the field list. During the run of the solver all valid field names are printed.
The output time can be changed too. Instead of writing at specific times in the simulation, you can also write after a certain number of time steps or depening on the wall clock time:

<pre>// write every 100th simulation time step
writeControl timeStep;
writeInterval 100;
</pre>

<pre>// every 3600 seconds of real wall clock time
writeControl runtime;
writeInterval 3600;
</pre>

If you use OpenFOAM before version 4.0 or 1606, the type of function object is:
<pre>
type writeRegisteredObject; // (instead of type writeObjects)
</pre>
If you use OpenFOAM before version 3.0, you have to load the library with
<pre>
functionObjectLibs ("libIOFunctionObjects.so"); // (instead of libs ( "libutilityFunctionObjects.so" ))
</pre>
and exchange the entry "writeControl" with "outputControl".

= OpenFOAM and ParaView on bwUniCluster=
ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/4.3.1</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
<pre>$ paraview</pre>
'''NOTICE:''' ParaView is a visualization software which requires X-server to run.

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.

----
[[Category:Engineering software]]
[[Category:BwUniCluster]]
[[Category:BwUniCluster_2.0]]
[[Category:BwForCluster_Chemistry]]
[[Category:BwForCluster_JUSTUS_2]]

BwUniCluster3.0/Software/OpenFoam

2018-06-25T16:20:10Z

A Saramet: /* Parallel run with OpenFOAM */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/openfoam
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]] | bwGRiD-Tübingen
|-
| License
| [https://www.openfoam.org/licence.php GNU General Public Licence]
|-
| Citing
| n/a
|-
| Links
| [https://www.openfoam.org/ Openfoam Homepage] | [https://www.openfoam.org/docs/ Documentation]
|-
| Graphical Interface
| No
|}
 
 
= Description =
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/cae/openfoam
|width=99%
|height=600
|border=0
}}
 
In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.
 
Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>
 

= Parallel run with OpenFOAM =
For a better performance on running OpenFOAM jobs in parallel on bwUniCluster, it is recommended to have the decomposed data in local folders on each node.

Therefore you may use *HPC scripts, wich will copy your data to the node specific folders after running the decomposePar, and copy it back to the local case folder before running reconstructPar.

Don't forget to allocate enough wall-time for decomposition and reconstruction of your cases. As the data will be processed directly on the nodes, and may be lost if the job is cancelled before the data is copied back into the case folder.

Following commands will do that for you:

<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>

instead of:
<pre>$ decomposePar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMesh -constant</pre>
 

For running jobs on multiple nodes, OpenFOAM needs passwordless communication between the nodes, to copy data into the local folders.

A small trick using ssh-keygen once will let your nodes to communicate freely over rsh.

Do it once (if you didn't do it already in the past):

<pre>
$ ssh-keygen
$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
</pre>

= Building an OpenFOAM batch file for parallel processing =
== General information ==
Before running OpenFOAM jobs in parallel, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use.

That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'', letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between.

There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results.

Decomposition and segments building process is handled bydecomposeParutility.

The number of subdomains, in which the geometry will be decomposed, is specified in "''system/decomposeParDict''", as well as the decomposition method to use.

The automatic decomposition method is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with not enough cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods.
 

== Wrapper script generation ==
'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposeParHPC &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable,
# in the header
mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel &&

reconstructParHPC

</source>
|}
 
'''Attention:''' The script above will run a parallel OpenFOAM Job with pre-installed OpenMPI. If you are using an OpenFOAM version wich comes with pre-installed Intel MPI (like, for examplecae/openfoam/v1712-impi) you will have to modify the batch script to use all the advantages of Intel MPI for parallel calculations. For details see:
* [[Batch_Jobs_-_bwUniCluster_Features|Batch Jobs Features]]

= Using I/O and reducing the amount of data and files =
In OpenFOAM, you can control which variables or fields are written at specific times. For example, for post-processing purposes, you might need only a subset of variables. In order to control which files will be written, there is a function object called "writeObjects".

An example controlDict file may look like this: At the top of the file (entry "writeControl") you specify that ALL fields (variables) required for restarting are saved every 12 wall-clock hours. Then, additionally, at the bottom of the controlDict in the "functions" block, you can add a function object of type "writeObjects". With this function object, you can control the output of specific fields independent of the entry at the top of the file:

{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="text">
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.1.x |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

startFrom latestTime;
startTime 0;
stopAt endTime;
endTime 1e2;
deltaT 1e-5;

writeControl clockTime;
writeInterval 43200; // write ALL fields necessary to restart your simulation
// every 43200 wall-clock seconds = 12 hours of real time

purgeWrite 0;
writeFormat binary;
writePrecision 10;
writeCompression off;
timeFormat general;
timePrecision 10;
runTimeModifiable false;

functions
{
writeFields // name of the function object
{
type writeObjects;
libs ( "libutilityFunctionObjects.so" );

objects
(
T U rho // list of fields/variables to be written
);

// E.g. write every 1e-5 seconds of simulation time only the specified fields
writeControl runTime;
writeInterval 1e-5; // write every 1e-5 seconds
}
}

</source>
|}
 

You can also define multiple function objects in order to write different subsets of fields at different times. You can also use wildcards in the list of fields- for example, in order to write out all fields starting with "RR_" you can add
<pre>
"RR_.*"
</pre>
to the list of objects. You can get a list of valid field names by writing "banana" in the field list. During the run of the solver all valid field names are printed.
The output time can be changed too. Instead of writing at specific times in the simulation, you can also write after a certain number of time steps or depening on the wall clock time:

<pre>// write every 100th simulation time step
writeControl timeStep;
writeInterval 100;
</pre>

<pre>// every 3600 seconds of real wall clock time
writeControl runtime;
writeInterval 3600;
</pre>

If you use OpenFOAM before version 4.0 or 1606, the type of function object is:
<pre>
type writeRegisteredObject; // (instead of type writeObjects)
</pre>
If you use OpenFOAM before version 3.0, you have to load the library with
<pre>
functionObjectLibs ("libIOFunctionObjects.so"); // (instead of libs ( "libutilityFunctionObjects.so" ))
</pre>
and exchange the entry "writeControl" with "outputControl".

= OpenFOAM and ParaView on bwUniCluster=
ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/4.3.1</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
<pre>$ paraview</pre>
'''NOTICE:''' ParaView is a visualization software which requires X-server to run.

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.

----
[[Category:Engineering software]][[Category:bwUniCluster]][[Category:bwForCluster_Chemistry]]

BwUniCluster3.0/Software/OpenFoam

2018-06-25T16:19:36Z

A Saramet: /* Parallel run with OpenFOAM */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/openfoam
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]] | bwGRiD-Tübingen
|-
| License
| [https://www.openfoam.org/licence.php GNU General Public Licence]
|-
| Citing
| n/a
|-
| Links
| [https://www.openfoam.org/ Openfoam Homepage] | [https://www.openfoam.org/docs/ Documentation]
|-
| Graphical Interface
| No
|}
 
 
= Description =
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/cae/openfoam
|width=99%
|height=600
|border=0
}}
 
In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.
 
Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>
 

= Parallel run with OpenFOAM =
For a better performance on running OpenFOAM jobs in parallel on bwUniCluster, it is recommended to have the decomposed data in local folders on each node.

Therefore you may use *HPC scripts, wich will copy your data to the node specific folders after running the decomposePar, and copy it back to the local case folder before running reconstructPar.

Don't forget to allocate enough wall-time for decomposition and reconstruction of your cases. As the data will be processed directly on the nodes, and may be lost if the job is cancelled before the data is copied back into the case folder.

Following commands will do that for you:

<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>

instead of:
<pre>$ decomposePar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMesh -constant</pre>
 

For running jobs on multiple nodes, OpenFOAM needs passwordless communication between the nodes, to copy data into the local folders.

A small trick using ssh-keygen once will let your nodes to communicate freely over rsh.

Do it once:

<pre>
$ ssh-keygen
$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
</pre>

= Building an OpenFOAM batch file for parallel processing =
== General information ==
Before running OpenFOAM jobs in parallel, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use.

That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'', letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between.

There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results.

Decomposition and segments building process is handled bydecomposeParutility.

The number of subdomains, in which the geometry will be decomposed, is specified in "''system/decomposeParDict''", as well as the decomposition method to use.

The automatic decomposition method is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with not enough cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods.
 

== Wrapper script generation ==
'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposeParHPC &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable,
# in the header
mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel &&

reconstructParHPC

</source>
|}
 
'''Attention:''' The script above will run a parallel OpenFOAM Job with pre-installed OpenMPI. If you are using an OpenFOAM version wich comes with pre-installed Intel MPI (like, for examplecae/openfoam/v1712-impi) you will have to modify the batch script to use all the advantages of Intel MPI for parallel calculations. For details see:
* [[Batch_Jobs_-_bwUniCluster_Features|Batch Jobs Features]]

= Using I/O and reducing the amount of data and files =
In OpenFOAM, you can control which variables or fields are written at specific times. For example, for post-processing purposes, you might need only a subset of variables. In order to control which files will be written, there is a function object called "writeObjects".

An example controlDict file may look like this: At the top of the file (entry "writeControl") you specify that ALL fields (variables) required for restarting are saved every 12 wall-clock hours. Then, additionally, at the bottom of the controlDict in the "functions" block, you can add a function object of type "writeObjects". With this function object, you can control the output of specific fields independent of the entry at the top of the file:

{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="text">
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.1.x |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

startFrom latestTime;
startTime 0;
stopAt endTime;
endTime 1e2;
deltaT 1e-5;

writeControl clockTime;
writeInterval 43200; // write ALL fields necessary to restart your simulation
// every 43200 wall-clock seconds = 12 hours of real time

purgeWrite 0;
writeFormat binary;
writePrecision 10;
writeCompression off;
timeFormat general;
timePrecision 10;
runTimeModifiable false;

functions
{
writeFields // name of the function object
{
type writeObjects;
libs ( "libutilityFunctionObjects.so" );

objects
(
T U rho // list of fields/variables to be written
);

// E.g. write every 1e-5 seconds of simulation time only the specified fields
writeControl runTime;
writeInterval 1e-5; // write every 1e-5 seconds
}
}

</source>
|}
 

You can also define multiple function objects in order to write different subsets of fields at different times. You can also use wildcards in the list of fields- for example, in order to write out all fields starting with "RR_" you can add
<pre>
"RR_.*"
</pre>
to the list of objects. You can get a list of valid field names by writing "banana" in the field list. During the run of the solver all valid field names are printed.
The output time can be changed too. Instead of writing at specific times in the simulation, you can also write after a certain number of time steps or depening on the wall clock time:

<pre>// write every 100th simulation time step
writeControl timeStep;
writeInterval 100;
</pre>

<pre>// every 3600 seconds of real wall clock time
writeControl runtime;
writeInterval 3600;
</pre>

If you use OpenFOAM before version 4.0 or 1606, the type of function object is:
<pre>
type writeRegisteredObject; // (instead of type writeObjects)
</pre>
If you use OpenFOAM before version 3.0, you have to load the library with
<pre>
functionObjectLibs ("libIOFunctionObjects.so"); // (instead of libs ( "libutilityFunctionObjects.so" ))
</pre>
and exchange the entry "writeControl" with "outputControl".

= OpenFOAM and ParaView on bwUniCluster=
ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/4.3.1</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
<pre>$ paraview</pre>
'''NOTICE:''' ParaView is a visualization software which requires X-server to run.

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.

----
[[Category:Engineering software]][[Category:bwUniCluster]][[Category:bwForCluster_Chemistry]]

BwUniCluster3.0/Software/OpenFoam

2018-06-25T15:37:09Z

A Saramet: /* General information */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/openfoam
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]] | bwGRiD-Tübingen
|-
| License
| [https://www.openfoam.org/licence.php GNU General Public Licence]
|-
| Citing
| n/a
|-
| Links
| [https://www.openfoam.org/ Openfoam Homepage] | [https://www.openfoam.org/docs/ Documentation]
|-
| Graphical Interface
| No
|}
 
 
= Description =
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/cae/openfoam
|width=99%
|height=600
|border=0
}}
 
In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.
 
Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>
 

= Parallel run with OpenFOAM =
For a better performance on running OpenFOAM jobs in parallel on bwUniCluster, it is recommended to have the decomposed data in local folders on each node.

Therefore you may use *HPC scripts, wich will copy your data to the node specific folders after running the decomposePar, and copy it back to the local case folder before running reconstructPar.

Don't forget to allocate enough wall-time for decomposition and reconstruction of your cases. As the data will be processed directly on the nodes, and may be lost if the job is cancelled before the data is copied back into the case folder.

Following commands will do that for you:

<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>

instead of:
<pre>$ decomposePar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMesh -constant</pre>
 

= Building an OpenFOAM batch file for parallel processing =
== General information ==
Before running OpenFOAM jobs in parallel, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use.

That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'', letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between.

There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results.

Decomposition and segments building process is handled bydecomposeParutility.

The number of subdomains, in which the geometry will be decomposed, is specified in "''system/decomposeParDict''", as well as the decomposition method to use.

The automatic decomposition method is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with not enough cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods.
 

== Wrapper script generation ==
'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposeParHPC &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable,
# in the header
mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel &&

reconstructParHPC

</source>
|}
 
'''Attention:''' The script above will run a parallel OpenFOAM Job with pre-installed OpenMPI. If you are using an OpenFOAM version wich comes with pre-installed Intel MPI (like, for examplecae/openfoam/v1712-impi) you will have to modify the batch script to use all the advantages of Intel MPI for parallel calculations. For details see:
* [[Batch_Jobs_-_bwUniCluster_Features|Batch Jobs Features]]

= Using I/O and reducing the amount of data and files =
In OpenFOAM, you can control which variables or fields are written at specific times. For example, for post-processing purposes, you might need only a subset of variables. In order to control which files will be written, there is a function object called "writeObjects".

An example controlDict file may look like this: At the top of the file (entry "writeControl") you specify that ALL fields (variables) required for restarting are saved every 12 wall-clock hours. Then, additionally, at the bottom of the controlDict in the "functions" block, you can add a function object of type "writeObjects". With this function object, you can control the output of specific fields independent of the entry at the top of the file:

{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="text">
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.1.x |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

startFrom latestTime;
startTime 0;
stopAt endTime;
endTime 1e2;
deltaT 1e-5;

writeControl clockTime;
writeInterval 43200; // write ALL fields necessary to restart your simulation
// every 43200 wall-clock seconds = 12 hours of real time

purgeWrite 0;
writeFormat binary;
writePrecision 10;
writeCompression off;
timeFormat general;
timePrecision 10;
runTimeModifiable false;

functions
{
writeFields // name of the function object
{
type writeObjects;
libs ( "libutilityFunctionObjects.so" );

objects
(
T U rho // list of fields/variables to be written
);

// E.g. write every 1e-5 seconds of simulation time only the specified fields
writeControl runTime;
writeInterval 1e-5; // write every 1e-5 seconds
}
}

</source>
|}
 

You can also define multiple function objects in order to write different subsets of fields at different times. You can also use wildcards in the list of fields- for example, in order to write out all fields starting with "RR_" you can add
<pre>
"RR_.*"
</pre>
to the list of objects. You can get a list of valid field names by writing "banana" in the field list. During the run of the solver all valid field names are printed.
The output time can be changed too. Instead of writing at specific times in the simulation, you can also write after a certain number of time steps or depening on the wall clock time:

<pre>// write every 100th simulation time step
writeControl timeStep;
writeInterval 100;
</pre>

<pre>// every 3600 seconds of real wall clock time
writeControl runtime;
writeInterval 3600;
</pre>

If you use OpenFOAM before version 4.0 or 1606, the type of function object is:
<pre>
type writeRegisteredObject; // (instead of type writeObjects)
</pre>
If you use OpenFOAM before version 3.0, you have to load the library with
<pre>
functionObjectLibs ("libIOFunctionObjects.so"); // (instead of libs ( "libutilityFunctionObjects.so" ))
</pre>
and exchange the entry "writeControl" with "outputControl".

= OpenFOAM and ParaView on bwUniCluster=
ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/4.3.1</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
<pre>$ paraview</pre>
'''NOTICE:''' ParaView is a visualization software which requires X-server to run.

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.

----
[[Category:Engineering software]][[Category:bwUniCluster]][[Category:bwForCluster_Chemistry]]

BwUniCluster3.0/Software/OpenFoam

2018-06-25T15:16:47Z

A Saramet: /* Wrapper script generation */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/openfoam
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]] | bwGRiD-Tübingen
|-
| License
| [https://www.openfoam.org/licence.php GNU General Public Licence]
|-
| Citing
| n/a
|-
| Links
| [https://www.openfoam.org/ Openfoam Homepage] | [https://www.openfoam.org/docs/ Documentation]
|-
| Graphical Interface
| No
|}
 
 
= Description =
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/cae/openfoam
|width=99%
|height=600
|border=0
}}
 
In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.
 
Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>
 

= Parallel run with OpenFOAM =
For a better performance on running OpenFOAM jobs in parallel on bwUniCluster, it is recommended to have the decomposed data in local folders on each node.

Therefore you may use *HPC scripts, wich will copy your data to the node specific folders after running the decomposePar, and copy it back to the local case folder before running reconstructPar.

Don't forget to allocate enough wall-time for decomposition and reconstruction of your cases. As the data will be processed directly on the nodes, and may be lost if the job is cancelled before the data is copied back into the case folder.

Following commands will do that for you:

<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>

instead of:
<pre>$ decomposePar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMesh -constant</pre>
 

= Building an OpenFOAM batch file for parallel processing =
== General information ==
To run any job in a parallel mode with OpenFOAM, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use. That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'' mode, letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between. There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results. Decomposition and segments building process is handled bydecomposeParutility. In "''system/decomposeParDict''" you may specify, how many "segments" you want your geometry domain to be divided in, and which decomposition method to use. The automatic one is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with few cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods. There are some other ways as well, with more documentation on the internet.
 
== Wrapper script generation ==
'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposeParHPC &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable,
# in the header
mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel &&

reconstructParHPC

</source>
|}
 
'''Attention:''' The script above will run a parallel OpenFOAM Job with pre-installed OpenMPI. If you are using an OpenFOAM version wich comes with pre-installed Intel MPI (like, for examplecae/openfoam/v1712-impi) you will have to modify the batch script to use all the advantages of Intel MPI for parallel calculations. For details see:
* [[Batch_Jobs_-_bwUniCluster_Features|Batch Jobs Features]]

= Using I/O and reducing the amount of data and files =
In OpenFOAM, you can control which variables or fields are written at specific times. For example, for post-processing purposes, you might need only a subset of variables. In order to control which files will be written, there is a function object called "writeObjects".

An example controlDict file may look like this: At the top of the file (entry "writeControl") you specify that ALL fields (variables) required for restarting are saved every 12 wall-clock hours. Then, additionally, at the bottom of the controlDict in the "functions" block, you can add a function object of type "writeObjects". With this function object, you can control the output of specific fields independent of the entry at the top of the file:

{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="text">
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.1.x |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

startFrom latestTime;
startTime 0;
stopAt endTime;
endTime 1e2;
deltaT 1e-5;

writeControl clockTime;
writeInterval 43200; // write ALL fields necessary to restart your simulation
// every 43200 wall-clock seconds = 12 hours of real time

purgeWrite 0;
writeFormat binary;
writePrecision 10;
writeCompression off;
timeFormat general;
timePrecision 10;
runTimeModifiable false;

functions
{
writeFields // name of the function object
{
type writeObjects;
libs ( "libutilityFunctionObjects.so" );

objects
(
T U rho // list of fields/variables to be written
);

// E.g. write every 1e-5 seconds of simulation time only the specified fields
writeControl runTime;
writeInterval 1e-5; // write every 1e-5 seconds
}
}

</source>
|}
 

You can also define multiple function objects in order to write different subsets of fields at different times. You can also use wildcards in the list of fields- for example, in order to write out all fields starting with "RR_" you can add
<pre>
"RR_.*"
</pre>
to the list of objects. You can get a list of valid field names by writing "banana" in the field list. During the run of the solver all valid field names are printed.
The output time can be changed too. Instead of writing at specific times in the simulation, you can also write after a certain number of time steps or depening on the wall clock time:

<pre>// write every 100th simulation time step
writeControl timeStep;
writeInterval 100;
</pre>

<pre>// every 3600 seconds of real wall clock time
writeControl runtime;
writeInterval 3600;
</pre>

If you use OpenFOAM before version 4.0 or 1606, the type of function object is:
<pre>
type writeRegisteredObject; // (instead of type writeObjects)
</pre>
If you use OpenFOAM before version 3.0, you have to load the library with
<pre>
functionObjectLibs ("libIOFunctionObjects.so"); // (instead of libs ( "libutilityFunctionObjects.so" ))
</pre>
and exchange the entry "writeControl" with "outputControl".

= OpenFOAM and ParaView on bwUniCluster=
ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/4.3.1</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
<pre>$ paraview</pre>
'''NOTICE:''' ParaView is a visualization software which requires X-server to run.

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.

----
[[Category:Engineering software]][[Category:bwUniCluster]][[Category:bwForCluster_Chemistry]]

BwUniCluster3.0/Software/OpenFoam

2018-06-25T15:15:15Z

A Saramet: /* Parallel run with OpenFOAM */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/openfoam
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]] | bwGRiD-Tübingen
|-
| License
| [https://www.openfoam.org/licence.php GNU General Public Licence]
|-
| Citing
| n/a
|-
| Links
| [https://www.openfoam.org/ Openfoam Homepage] | [https://www.openfoam.org/docs/ Documentation]
|-
| Graphical Interface
| No
|}
 
 
= Description =
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/cae/openfoam
|width=99%
|height=600
|border=0
}}
 
In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.
 
Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>
 

= Parallel run with OpenFOAM =
For a better performance on running OpenFOAM jobs in parallel on bwUniCluster, it is recommended to have the decomposed data in local folders on each node.

Therefore you may use *HPC scripts, wich will copy your data to the node specific folders after running the decomposePar, and copy it back to the local case folder before running reconstructPar.

Don't forget to allocate enough wall-time for decomposition and reconstruction of your cases. As the data will be processed directly on the nodes, and may be lost if the job is cancelled before the data is copied back into the case folder.

Following commands will do that for you:

<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>

instead of:
<pre>$ decomposePar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpirun --bind-to core --map-by core -report-bindings snappyHexMesh -overwrite -parallel
$ reconstructParMesh -constant</pre>
 

= Building an OpenFOAM batch file for parallel processing =
== General information ==
To run any job in a parallel mode with OpenFOAM, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use. That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'' mode, letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between. There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results. Decomposition and segments building process is handled bydecomposeParutility. In "''system/decomposeParDict''" you may specify, how many "segments" you want your geometry domain to be divided in, and which decomposition method to use. The automatic one is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with few cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods. There are some other ways as well, with more documentation on the internet.
 
== Wrapper script generation ==
'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposePar &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable,
# in the header
mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel &&

# remove reconstructPar and the '&&' operator from the command above
# if you would like to reconstruct the case later
reconstructPar

</source>
|}
 
'''Attention:''' The script above will run a parallel OpenFOAM Job with pre-installed OpenMPI. If you are using an OpenFOAM version wich comes with pre-installed Intel MPI (like, for examplecae/openfoam/v1712-impi) you will have to modify the batch script to use all the advantages of Intel MPI for parallel calculations. For details see:
* [[Batch_Jobs_-_bwUniCluster_Features|Batch Jobs Features]]

= Using I/O and reducing the amount of data and files =
In OpenFOAM, you can control which variables or fields are written at specific times. For example, for post-processing purposes, you might need only a subset of variables. In order to control which files will be written, there is a function object called "writeObjects".

An example controlDict file may look like this: At the top of the file (entry "writeControl") you specify that ALL fields (variables) required for restarting are saved every 12 wall-clock hours. Then, additionally, at the bottom of the controlDict in the "functions" block, you can add a function object of type "writeObjects". With this function object, you can control the output of specific fields independent of the entry at the top of the file:

{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="text">
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.1.x |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

startFrom latestTime;
startTime 0;
stopAt endTime;
endTime 1e2;
deltaT 1e-5;

writeControl clockTime;
writeInterval 43200; // write ALL fields necessary to restart your simulation
// every 43200 wall-clock seconds = 12 hours of real time

purgeWrite 0;
writeFormat binary;
writePrecision 10;
writeCompression off;
timeFormat general;
timePrecision 10;
runTimeModifiable false;

functions
{
writeFields // name of the function object
{
type writeObjects;
libs ( "libutilityFunctionObjects.so" );

objects
(
T U rho // list of fields/variables to be written
);

// E.g. write every 1e-5 seconds of simulation time only the specified fields
writeControl runTime;
writeInterval 1e-5; // write every 1e-5 seconds
}
}

</source>
|}
 

You can also define multiple function objects in order to write different subsets of fields at different times. You can also use wildcards in the list of fields- for example, in order to write out all fields starting with "RR_" you can add
<pre>
"RR_.*"
</pre>
to the list of objects. You can get a list of valid field names by writing "banana" in the field list. During the run of the solver all valid field names are printed.
The output time can be changed too. Instead of writing at specific times in the simulation, you can also write after a certain number of time steps or depening on the wall clock time:

<pre>// write every 100th simulation time step
writeControl timeStep;
writeInterval 100;
</pre>

<pre>// every 3600 seconds of real wall clock time
writeControl runtime;
writeInterval 3600;
</pre>

If you use OpenFOAM before version 4.0 or 1606, the type of function object is:
<pre>
type writeRegisteredObject; // (instead of type writeObjects)
</pre>
If you use OpenFOAM before version 3.0, you have to load the library with
<pre>
functionObjectLibs ("libIOFunctionObjects.so"); // (instead of libs ( "libutilityFunctionObjects.so" ))
</pre>
and exchange the entry "writeControl" with "outputControl".

= OpenFOAM and ParaView on bwUniCluster=
ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/4.3.1</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
<pre>$ paraview</pre>
'''NOTICE:''' ParaView is a visualization software which requires X-server to run.

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.

----
[[Category:Engineering software]][[Category:bwUniCluster]][[Category:bwForCluster_Chemistry]]

BwUniCluster3.0/Software/OpenFoam

2018-03-19T14:16:46Z

A Saramet: /* Wrapper script generation */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/openfoam
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]] | bwGRiD-Tübingen
|-
| License
| [https://www.openfoam.org/licence.php GNU General Public Licence]
|-
| Citing
| n/a
|-
| Links
| [https://www.openfoam.org/ Openfoam Homepage] | [https://www.openfoam.org/docs/ Documentation]
|-
| Graphical Interface
| No
|}
 
 
= Description =
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/cae/openfoam
|width=99%
|height=600
|border=0
}}
 
In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.
 
Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>
 

= Parallel run with OpenFOAM =
To improve the concurrently solving process and decrease the error occurrence probability while running an OpenFOAM job in parallel (''on a singlenode''), some modifications have been introduced. Specifically, after the case decomposition is done, it is recommended to save the decomposed data directly on the nodes in a local folder and it run from there. When the calculations are over, the data is moved back to the case folder and reconstructed. It will improve the overall performance, considering that you allocated enough wall-time to decompose and rebuild your cases, as it moves theprocessor*folders to and out of the nodes local work-space. For such a procedure it is necessary to use the following commands for decomposition and reconstruction of the geometry domain:
<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>
instead of:
<pre>$ decomposePar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMesh -constant</pre>
 

= Building an OpenFOAM batch file for parallel processing =
== General information ==
To run any job in a parallel mode with OpenFOAM, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use. That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'' mode, letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between. There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results. Decomposition and segments building process is handled bydecomposeParutility. In "''system/decomposeParDict''" you may specify, how many "segments" you want your geometry domain to be divided in, and which decomposition method to use. The automatic one is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with few cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods. There are some other ways as well, with more documentation on the internet.
 
== Wrapper script generation ==
'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposePar &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable,
# in the header
mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel &&

# remove reconstructPar and the '&&' operator from the command above
# if you would like to reconstruct the case later
reconstructPar

</source>
|}
 
'''Attention:''' The script above will run a parallel OpenFOAM Job with pre-installed OpenMPI. If you are using an OpenFOAM version wich comes with pre-installed Intel MPI (like, for examplecae/openfoam/v1712-impi) you will have to modify the batch script to use all the advantages of Intel MPI for parallel calculations. For details see:
* [[Batch_Jobs_-_bwUniCluster_Features|Batch Jobs Features]]

= Using I/O and reducing the amount of data and files =
In OpenFOAM, you can control which variables or fields are written at specific times. For example, for post-processing purposes, you might need only a subset of variables. In order to control which files will be written, there is a function object called "writeObjects".

An example controlDict file may look like this: At the top of the file (entry "writeControl") you specify that ALL fields (variables) required for restarting are saved every 12 wall-clock hours. Then, additionally, at the bottom of the controlDict in the "functions" block, you can add a function object of type "writeObjects". With this function object, you can control the output of specific fields independent of the entry at the top of the file:

{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="text">
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.1.x |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

startFrom latestTime;
startTime 0;
stopAt endTime;
endTime 1e2;
deltaT 1e-5;

writeControl clockTime;
writeInterval 43200; // write ALL fields necessary to restart your simulation
// every 43200 wall-clock seconds = 12 hours of real time

purgeWrite 0;
writeFormat binary;
writePrecision 10;
writeCompression off;
timeFormat general;
timePrecision 10;
runTimeModifiable false;

functions
{
writeFields // name of the function object
{
type writeObjects;
libs ( "libutilityFunctionObjects.so" );

objects
(
T U rho // list of fields/variables to be written
);

// E.g. write every 1e-5 seconds of simulation time only the specified fields
writeControl runTime;
writeInterval 1e-5; // write every 1e-5 seconds
}
}

</source>
|}
 

You can also define multiple function objects in order to write different subsets of fields at different times. You can also use wildcards in the list of fields- for example, in order to write out all fields starting with "RR_" you can add
<pre>
"RR_.*"
</pre>
to the list of objects. You can get a list of valid field names by writing "banana" in the field list. During the run of the solver all valid field names are printed.
The output time can be changed too. Instead of writing at specific times in the simulation, you can also write after a certain number of time steps or depening on the wall clock time:

<pre>// write every 100th simulation time step
writeControl timeStep;
writeInterval 100;
</pre>

<pre>// every 3600 seconds of real wall clock time
writeControl runtime;
writeInterval 3600;
</pre>

If you use OpenFOAM before version 4.0 or 1606, the type of function object is:
<pre>
type writeRegisteredObject; // (instead of type writeObjects)
</pre>
If you use OpenFOAM before version 3.0, you have to load the library with
<pre>
functionObjectLibs ("libIOFunctionObjects.so"); // (instead of libs ( "libutilityFunctionObjects.so" ))
</pre>
and exchange the entry "writeControl" with "outputControl".

= OpenFOAM and ParaView on bwUniCluster=
ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/4.3.1</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
<pre>$ paraview</pre>
'''NOTICE:''' ParaView is a visualization software which requires X-server to run.

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.

----
[[Category:Engineering software]][[Category:bwUniCluster]][[Category:bwForCluster_Chemistry]]

BwUniCluster3.0/Software/OpenFoam

2017-11-03T10:18:53Z

A Saramet: /* Wrapper script generation */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/openfoam
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]] | bwGRiD-Tübingen
|-
| License
| [http://www.openfoam.org/licence.php GNU General Public Licence]
|-
| Citing
| n/a
|-
| Links
| [http://www.openfoam.org/ Openfoam Homepage] | [http://www.openfoam.org/docs/ Documentation]
|-
| Graphical Interface
| No
|}
 
 
= Description =
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/cae/openfoam
|width=99%
|height=600
|border=0
}}
 
In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.
 
Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>
 

= Parallel run with OpenFOAM =
To improve the concurrently solving process and decrease the error occurrence probability while running an OpenFOAM job in parallel (''on a singlenode''), some modifications have been introduced. Specifically, after the case decomposition is done, it is recommended to save the decomposed data directly on the nodes in a local folder and it run from there. When the calculations are over, the data is moved back to the case folder and reconstructed. It will improve the overall performance, considering that you allocated enough wall-time to decompose and rebuild your cases, as it moves theprocessor*folders to and out of the nodes local work-space. For such a procedure it is necessary to use the following commands for decomposition and reconstruction of the geometry domain:
<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>
instead of:
<pre>$ decomposePar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMesh -constant</pre>
 

= Building an OpenFOAM batch file for parallel processing =
== General information ==
To run any job in a parallel mode with OpenFOAM, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use. That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'' mode, letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between. There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results. Decomposition and segments building process is handled bydecomposeParutility. In "''system/decomposeParDict''" you may specify, how many "segments" you want your geometry domain to be divided in, and which decomposition method to use. The automatic one is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with few cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods. There are some other ways as well, with more documentation on the internet.
 
== Wrapper script generation ==
'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposePar &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable,
# in the header
mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel &&

# remove reconstructPar and the '&&' operator from the command above
# if you would like to reconstruct the case later
reconstructPar

</source>
|}
 

= Using I/O and reducing the amount of data and files =
In OpenFOAM, you can control which variables or fields are written at specific times. For example, for post-processing purposes, you might need only a subset of variables. In order to control which files will be written, there is a function object called "writeObjects".

An example controlDict file may look like this: At the top of the file (entry "writeControl") you specify that ALL fields (variables) required for restarting are saved every 12 wall-clock hours. Then, additionally, at the bottom of the controlDict in the "functions" block, you can add a function object of type "writeObjects". With this function object, you can control the output of specific fields independent of the entry at the top of the file:

{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="text">
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.1.x |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

startFrom latestTime;
startTime 0;
stopAt endTime;
endTime 1e2;
deltaT 1e-5;

writeControl clockTime;
writeInterval 43200; // write ALL fields necessary to restart your simulation
// every 43200 wall-clock seconds = 12 hours of real time

purgeWrite 0;
writeFormat binary;
writePrecision 10;
writeCompression off;
timeFormat general;
timePrecision 10;
runTimeModifiable false;

functions
{
writeFields // name of the function object
{
type writeObjects;
libs ( "libutilityFunctionObjects.so" );

objects
(
T U rho // list of fields/variables to be written
);

// E.g. write every 1e-5 seconds of simulation time only the specified fields
writeControl runTime;
writeInterval 1e-5; // write every 1e-5 seconds
}
}

</source>
|}
 

You can also define multiple function objects in order to write different subsets of fields at different times. You can also use wildcards in the list of fields- for example, in order to write out all fields starting with "RR_" you can add
<pre>
"RR_.*"
</pre>
to the list of objects. You can get a list of valid field names by writing "banana" in the field list. During the run of the solver all valid field names are printed.
The output time can be changed too. Instead of writing at specific times in the simulation, you can also write after a certain number of time steps or depening on the wall clock time:

<pre>// write every 100th simulation time step
writeControl timeStep;
writeInterval 100;
</pre>

<pre>// every 3600 seconds of real wall clock time
writeControl runtime;
writeInterval 3600;
</pre>

If you use OpenFOAM before version 4.0 or 1606, the type of function object is:
<pre>
type writeRegisteredObject; // (instead of type writeObjects)
</pre>
If you use OpenFOAM before version 3.0, you have to load the library with
<pre>
functionObjectLibs ("libIOFunctionObjects.so"); // (instead of libs ( "libutilityFunctionObjects.so" ))
</pre>
and exchange the entry "writeControl" with "outputControl".

= OpenFOAM and ParaView on bwUniCluster=
ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/4.3.1</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
<pre>$ paraview</pre>
'''NOTICE:''' ParaView is a visualization software which requires X-server to run.

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.

----
[[Category:Engineering software]][[Category:bwUniCluster]][[Category:bwForCluster_Chemistry]]

BwUniCluster3.0/Software/OpenFoam

2017-04-11T15:03:57Z

A Saramet: /* Parallel run with OpenFOAM */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/openfoam
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]] | bwGRiD-Tübingen
|-
| License
| [http://www.openfoam.org/licence.php GNU General Public Licence]
|-
| Citing
| n/a
|-
| Links
| [http://www.openfoam.org/ Openfoam Homepage] | [http://www.openfoam.org/docs/ Documentation]
|-
| Graphical Interface
| No
|}
 
 
= Description =
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/cae/openfoam
|width=99%
|height=600
|border=0
}}
 
In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.
 
Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>
 

= Parallel run with OpenFOAM =
To improve the concurrently solving process and decrease the error occurrence probability while running an OpenFOAM job in parallel (''on a singlenode''), some modifications have been introduced. Specifically, after the case decomposition is done, it is recommended to save the decomposed data directly on the nodes in a local folder and it run from there. When the calculations are over, the data is moved back to the case folder and reconstructed. It will improve the overall performance, considering that you allocated enough wall-time to decompose and rebuild your cases, as it moves theprocessor*folders to and out of the nodes local work-space. For such a procedure it is necessary to use the following commands for decomposition and reconstruction of the geometry domain:
<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>
instead of:
<pre>$ decomposePar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMesh -constant</pre>
 

= Building an OpenFOAM batch file for parallel processing =
== General information ==
To run any job in a parallel mode with OpenFOAM, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use. That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'' mode, letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between. There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results. Decomposition and segments building process is handled bydecomposeParutility. In "''system/decomposeParDict''" you may specify, how many "segments" you want your geometry domain to be divided in, and which decomposition method to use. The automatic one is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with few cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods. There are some other ways as well, with more documentation on the internet.
 
== Wrapper script generation ==
'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

startexe="mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel"

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposePar &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable,
# in the header
echo $startexe
exec $startexe &&

# remove reconstructPar and the '&&' operator from the command above
# if you would like to reconstruct the case later
reconstructPar

</source>
|}
 

= Using I/O and reducing the amount of data and files =
In OpenFOAM, you can control which variables or fields are written at specific times. For example, for post-processing purposes, you might need only a subset of variables. In order to control which files will be written, there is a function object called "writeObjects".

An example controlDict file may look like this: At the top of the file (entry "writeControl") you specify that ALL fields (variables) required for restarting are saved every 12 wall-clock hours. Then, additionally, at the bottom of the controlDict in the "functions" block, you can add a function object of type "writeObjects". With this function object, you can control the output of specific fields independent of the entry at the top of the file:

{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="text">
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.1.x |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

startFrom latestTime;
startTime 0;
stopAt endTime;
endTime 1e2;
deltaT 1e-5;

writeControl clockTime;
writeInterval 43200; // write ALL fields necessary to restart your simulation
// every 43200 wall-clock seconds = 12 hours of real time

purgeWrite 0;
writeFormat binary;
writePrecision 10;
writeCompression off;
timeFormat general;
timePrecision 10;
runTimeModifiable false;

functions
{
writeFields // name of the function object
{
type writeObjects;
libs ( "libutilityFunctionObjects.so" );

objects
(
T U rho // list of fields/variables to be written
);

// E.g. write every 1e-5 seconds of simulation time only the specified fields
writeControl runTime;
writeInterval 1e-5; // write every 1e-5 seconds
}
}

</source>
|}
 

You can also define multiple function objects in order to write different subsets of fields at different times. You can also use wildcards in the list of fields- for example, in order to write out all fields starting with "RR_" you can add
<pre>
"RR_.*"
</pre>
to the list of objects. You can get a list of valid field names by writing "banana" in the field list. During the run of the solver all valid field names are printed.
The output time can be changed too. Instead of writing at specific times in the simulation, you can also write after a certain number of time steps or depening on the wall clock time:

<pre>// write every 100th simulation time step
writeControl timeStep;
writeInterval 100;
</pre>

<pre>// every 3600 seconds of real wall clock time
writeControl runtime;
writeInterval 3600;
</pre>

If you use OpenFOAM before version 4.0 or 1606, the type of function object is:
<pre>
type writeRegisteredObject; // (instead of type writeObjects)
</pre>
If you use OpenFOAM before version 3.0, you have to load the library with
<pre>
functionObjectLibs ("libIOFunctionObjects.so"); // (instead of libs ( "libutilityFunctionObjects.so" ))
</pre>
and exchange the entry "writeControl" with "outputControl".

= OpenFOAM and ParaView on bwUniCluster=
ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/4.3.1</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
<pre>$ paraview</pre>
'''NOTICE:''' ParaView is a visualization software which requires X-server to run.

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.

----
[[Category:Engineering software]][[Category:bwUniCluster]][[Category:bwForCluster_Chemistry]]

BwUniCluster3.0/Software/OpenFoam

2017-04-11T15:03:09Z

A Saramet: /* Parallel run with OpenFOAM */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/openfoam
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]] | bwGRiD-Tübingen
|-
| License
| [http://www.openfoam.org/licence.php GNU General Public Licence]
|-
| Citing
| n/a
|-
| Links
| [http://www.openfoam.org/ Openfoam Homepage] | [http://www.openfoam.org/docs/ Documentation]
|-
| Graphical Interface
| No
|}
 
 
= Description =
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/cae/openfoam
|width=99%
|height=600
|border=0
}}
 
In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.
 
Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>
 

= Parallel run with OpenFOAM =
To improve the concurrently solving process and decrease the error occurrence probability while running an OpenFOAM job in parallel (''on a singlenode''), some modifications have been introduced. Specifically, after the case decomposition is done, it is recommended to save the decomposed data directly on the nodes local folder and run from there. When the calculations are over, the data is moved back to the case folder and reconstructed. It will improve the overall performance, considering that you allocated enough wall-time to decompose and rebuild your cases, as it moves theprocessor*folders to and out of the nodes local work-space. For such a procedure it is necessary to use the following commands for decomposition and reconstruction of the geometry domain:
<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>
instead of:
<pre>$ decomposePar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMesh -constant</pre>
 

= Building an OpenFOAM batch file for parallel processing =
== General information ==
To run any job in a parallel mode with OpenFOAM, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use. That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'' mode, letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between. There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results. Decomposition and segments building process is handled bydecomposeParutility. In "''system/decomposeParDict''" you may specify, how many "segments" you want your geometry domain to be divided in, and which decomposition method to use. The automatic one is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with few cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods. There are some other ways as well, with more documentation on the internet.
 
== Wrapper script generation ==
'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

startexe="mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel"

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposePar &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable,
# in the header
echo $startexe
exec $startexe &&

# remove reconstructPar and the '&&' operator from the command above
# if you would like to reconstruct the case later
reconstructPar

</source>
|}
 

= Using I/O and reducing the amount of data and files =
In OpenFOAM, you can control which variables or fields are written at specific times. For example, for post-processing purposes, you might need only a subset of variables. In order to control which files will be written, there is a function object called "writeObjects".

An example controlDict file may look like this: At the top of the file (entry "writeControl") you specify that ALL fields (variables) required for restarting are saved every 12 wall-clock hours. Then, additionally, at the bottom of the controlDict in the "functions" block, you can add a function object of type "writeObjects". With this function object, you can control the output of specific fields independent of the entry at the top of the file:

{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="text">
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.1.x |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

startFrom latestTime;
startTime 0;
stopAt endTime;
endTime 1e2;
deltaT 1e-5;

writeControl clockTime;
writeInterval 43200; // write ALL fields necessary to restart your simulation
// every 43200 wall-clock seconds = 12 hours of real time

purgeWrite 0;
writeFormat binary;
writePrecision 10;
writeCompression off;
timeFormat general;
timePrecision 10;
runTimeModifiable false;

functions
{
writeFields // name of the function object
{
type writeObjects;
libs ( "libutilityFunctionObjects.so" );

objects
(
T U rho // list of fields/variables to be written
);

// E.g. write every 1e-5 seconds of simulation time only the specified fields
writeControl runTime;
writeInterval 1e-5; // write every 1e-5 seconds
}
}

</source>
|}
 

You can also define multiple function objects in order to write different subsets of fields at different times. You can also use wildcards in the list of fields- for example, in order to write out all fields starting with "RR_" you can add
<pre>
"RR_.*"
</pre>
to the list of objects. You can get a list of valid field names by writing "banana" in the field list. During the run of the solver all valid field names are printed.
The output time can be changed too. Instead of writing at specific times in the simulation, you can also write after a certain number of time steps or depening on the wall clock time:

<pre>// write every 100th simulation time step
writeControl timeStep;
writeInterval 100;
</pre>

<pre>// every 3600 seconds of real wall clock time
writeControl runtime;
writeInterval 3600;
</pre>

If you use OpenFOAM before version 4.0 or 1606, the type of function object is:
<pre>
type writeRegisteredObject; // (instead of type writeObjects)
</pre>
If you use OpenFOAM before version 3.0, you have to load the library with
<pre>
functionObjectLibs ("libIOFunctionObjects.so"); // (instead of libs ( "libutilityFunctionObjects.so" ))
</pre>
and exchange the entry "writeControl" with "outputControl".

= OpenFOAM and ParaView on bwUniCluster=
ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/4.3.1</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
<pre>$ paraview</pre>
'''NOTICE:''' ParaView is a visualization software which requires X-server to run.

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.

----
[[Category:Engineering software]][[Category:bwUniCluster]][[Category:bwForCluster_Chemistry]]

BwUniCluster3.0/Software/OpenFoam

2017-04-11T13:44:27Z

A Saramet: /* OpenFOAM data handling on bwUniCluster */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/openfoam
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]] | bwGRiD-Tübingen
|-
| License
| [http://www.openfoam.org/licence.php GNU General Public Licence]
|-
| Citing
| n/a
|-
| Links
| [http://www.openfoam.org/ Openfoam Homepage] | [http://www.openfoam.org/docs/ Documentation]
|-
| Graphical Interface
| No
|}
 
 
= Description =
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/cae/openfoam
|width=99%
|height=600
|border=0
}}
 
In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.
 
Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>
 

= Parallel run with OpenFOAM =
To improve the concurrently solving process and decrease the error occurrence probability while running an OpenFOAM job in parallel (''on a singlenode''), some modifications have been introduced. Specifically, after the case decomposition is done, it is recommended to save the decomposed data directly on the nodes local folder and run from there. When the calculations are over, the data is moved back to the case folder and reconstructed. It will improve the overall performance, considering that you allocated enough wall-time to decompose and rebuild your cases, as it moves theprocessor*folders to and out of the nodes local work-space. For such a procedure it is necessary to use the following commands for decomposition and reconstruction of the geometry domain:
<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>
instead of:
<pre>$ decomposPar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMesh -constant</pre>
 

= Building an OpenFOAM batch file for parallel processing =
== General information ==
To run any job in a parallel mode with OpenFOAM, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use. That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'' mode, letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between. There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results. Decomposition and segments building process is handled bydecomposeParutility. In "''system/decomposeParDict''" you may specify, how many "segments" you want your geometry domain to be divided in, and which decomposition method to use. The automatic one is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with few cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods. There are some other ways as well, with more documentation on the internet.
 
== Wrapper script generation ==
'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

startexe="mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel"

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposePar &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable,
# in the header
echo $startexe
exec $startexe &&

# remove reconstructPar and the '&&' operator from the command above
# if you would like to reconstruct the case later
reconstructPar

</source>
|}
 

= Using I/O and reducing the amount of data and files =
In OpenFOAM, you can control which variables or fields are written at specific times. For example, for post-processing purposes, you might need only a subset of variables. In order to control which files will be written, there is a function object called "writeObjects".

An example controlDict file may look like this: At the top of the file (entry "writeControl") you specify that ALL fields (variables) required for restarting are saved every 12 wall-clock hours. Then, additionally, at the bottom of the controlDict in the "functions" block, you can add a function object of type "writeObjects". With this function object, you can control the output of specific fields independent of the entry at the top of the file:

{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="text">
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.1.x |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

startFrom latestTime;
startTime 0;
stopAt endTime;
endTime 1e2;
deltaT 1e-5;

writeControl clockTime;
writeInterval 43200; // write ALL fields necessary to restart your simulation
// every 43200 wall-clock seconds = 12 hours of real time

purgeWrite 0;
writeFormat binary;
writePrecision 10;
writeCompression off;
timeFormat general;
timePrecision 10;
runTimeModifiable false;

functions
{
writeFields // name of the function object
{
type writeObjects;
libs ( "libutilityFunctionObjects.so" );

objects
(
T U rho // list of fields/variables to be written
);

// E.g. write every 1e-5 seconds of simulation time only the specified fields
writeControl runTime;
writeInterval 1e-5; // write every 1e-5 seconds
}
}

</source>
|}
 

You can also define multiple function objects in order to write different subsets of fields at different times. You can also use wildcards in the list of fields- for example, in order to write out all fields starting with "RR_" you can add
<pre>
"RR_.*"
</pre>
to the list of objects. You can get a list of valid field names by writing "banana" in the field list. During the run of the solver all valid field names are printed.
The output time can be changed too. Instead of writing at specific times in the simulation, you can also write after a certain number of time steps or depening on the wall clock time:

<pre>// write every 100th simulation time step
writeControl timeStep;
writeInterval 100;
</pre>

<pre>// every 3600 seconds of real wall clock time
writeControl runtime;
writeInterval 3600;
</pre>

If you use OpenFOAM before version 4.0 or 1606, the type of function object is:
<pre>
type writeRegisteredObject; // (instead of type writeObjects)
</pre>
If you use OpenFOAM before version 3.0, you have to load the library with
<pre>
functionObjectLibs ("libIOFunctionObjects.so"); // (instead of libs ( "libutilityFunctionObjects.so" ))
</pre>
and exchange the entry "writeControl" with "outputControl".

= OpenFOAM and ParaView on bwUniCluster=
ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/4.3.1</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
<pre>$ paraview</pre>
'''NOTICE:''' ParaView is a visualization software which requires X-server to run.

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.

----
[[Category:Engineering software]][[Category:bwUniCluster]][[Category:bwForCluster_Chemistry]]

BwUniCluster3.0/Software/OpenFoam

2017-04-11T11:37:16Z

A Saramet: /* Improving parallel run performance */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/openfoam
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]] | bwGRiD-Tübingen
|-
| License
| [http://www.openfoam.org/licence.php GNU General Public Licence]
|-
| Citing
| n/a
|-
| Links
| [http://www.openfoam.org/ Openfoam Homepage] | [http://www.openfoam.org/docs/ Documentation]
|-
| Graphical Interface
| No
|}
 
 
= Description =
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/cae/openfoam
|width=99%
|height=600
|border=0
}}
 
In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.
 
Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>
 

= OpenFOAM data handling on bwUniCluster =
OpenFOAM cases may contain huge amount of data. Therefore, it it recommended to run jobs in $WORK folder or using Workspaces, as described in [http://www.bwhpc-c5.de/wiki/index.php/BwUniCluster_Hardware_and_Architecture#File_Systems File Systems]

Self - compiled solvers and applications, on the other hand, should be installed in $HOME directory, as a regular backup of this folder is automatically done, saving your data and not requireing future recompilations.

Don't forget to load the OpenFOAM version before compiling your solver, as it will automatically load version configured compiler and mpi.

= Building an OpenFOAM batch file for parallel processing =
== General information ==
To run any job in a parallel mode with OpenFOAM, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use. That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'' mode, letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between. There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results. Decomposition and segments building process is handled bydecomposeParutility. In "''system/decomposeParDict''" you may specify, how many "segments" you want your geometry domain to be divided in, and which decomposition method to use. The automatic one is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with few cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods. There are some other ways as well, with more documentation on the internet.
 
== Wrapper script generation ==
'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

startexe="mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel"

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposePar &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable,
# in the header
echo $startexe
exec $startexe &&

# remove reconstructPar and the '&&' operator from the command above
# if you would like to reconstruct the case later
reconstructPar

</source>
|}
 

= Using I/O and reducing the amount of data and files =
In OpenFOAM, you can control which variables or fields are written at specific times. For example, for post-processing purposes, you might need only a subset of variables. In order to control which files will be written, there is a function object called "writeObjects".

An example controlDict file may look like this: At the top of the file (entry "writeControl") you specify that ALL fields (variables) required for restarting are saved every 12 wall-clock hours. Then, additionally, at the bottom of the controlDict in the "functions" block, you can add a function object of type "writeObjects". With this function object, you can control the output of specific fields independent of the entry at the top of the file:

{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="text">
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.1.x |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

startFrom latestTime;
startTime 0;
stopAt endTime;
endTime 1e2;
deltaT 1e-5;

writeControl clockTime;
writeInterval 43200; // write ALL fields necessary to restart your simulation
// every 43200 wall-clock seconds = 12 hours of real time

purgeWrite 0;
writeFormat binary;
writePrecision 10;
writeCompression off;
timeFormat general;
timePrecision 10;
runTimeModifiable false;

functions
{
writeFields // name of the function object
{
type writeObjects;
libs ( "libutilityFunctionObjects.so" );

objects
(
T U rho // list of fields/variables to be written
);

// E.g. write every 1e-5 seconds of simulation time only the specified fields
writeControl runTime;
writeInterval 1e-5; // write every 1e-5 seconds
}
}

</source>
|}
 

You can also define multiple function objects in order to write different subsets of fields at different times. You can also use wildcards in the list of fields- for example, in order to write out all fields starting with "RR_" you can add
<pre>
"RR_.*"
</pre>
to the list of objects. You can get a list of valid field names by writing "banana" in the field list. During the run of the solver all valid field names are printed.
The output time can be changed too. Instead of writing at specific times in the simulation, you can also write after a certain number of time steps or depening on the wall clock time:

<pre>// write every 100th simulation time step
writeControl timeStep;
writeInterval 100;
</pre>

<pre>// every 3600 seconds of real wall clock time
writeControl runtime;
writeInterval 3600;
</pre>

If you use OpenFOAM before version 4.0 or 1606, the type of function object is:
<pre>
type writeRegisteredObject; // (instead of type writeObjects)
</pre>
If you use OpenFOAM before version 3.0, you have to load the library with
<pre>
functionObjectLibs ("libIOFunctionObjects.so"); // (instead of libs ( "libutilityFunctionObjects.so" ))
</pre>
and exchange the entry "writeControl" with "outputControl".

= OpenFOAM and ParaView on bwUniCluster=
ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/4.3.1</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
<pre>$ paraview</pre>
'''NOTICE:''' ParaView is a visualization software which requires X-server to run.

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.

----
[[Category:Engineering software]][[Category:bwUniCluster]][[Category:bwForCluster_Chemistry]]

BwUniCluster3.0/Software/OpenFoam

2015-12-17T13:10:25Z

A Saramet: /* Wrapper script generation */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/openfoam
|-
| Availability
| [[bwUniCluster]]
|-
| License
| [http://www.openfoam.org/licence.php GNU General Public Licence]
|-
| Citing
| n/a
|-
| Links
| [http://www.openfoam.org/ Openfoam Homepage] | [http://www.openfoam.org/docs/ Documentation]
|-
| Graphical Interface
| No
|}
 
 
= Description =
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/cae/openfoam
|width=99%
|height=450
|border=0
}}
 
In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.
 
Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>
 

= Improving parallel run performance =
To improve the concurrently solving process and decrease the error occurrence probability while running an OpenFOAM job in parallel (''on a singlenode''), some modifications have been introduced. Specifically, after the case decomposition is done, it is recommended to save the decomposed data directly on the nodes in the pre-allocated work-space and use it from there. When the calculations are over, the data is moved back to the case folder and reconstructed. It will improve the overall performance, considering that you allocated enough wall-time to decompose and rebuild your cases, as it moves theprocessor*folders to and out of the nodes local work-space. For such a procedure it is necessary to use the following commands for decomposition and reconstruction of the geometry domain:
<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>
instead of:
<pre>$ decomposPar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMesh -constant</pre>
 
= Building an OpenFOAM batch file for parallel processing =
== General information ==
To run any job in a parallel mode with OpenFOAM, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use. That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'' mode, letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between. There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results. Decomposition and segments building process is handled bydecomposeParutility. In "''system/decomposeParDict''" you may specify, how many "segments" you want your geometry domain to be divided in, and which decomposition method to use. The automatic one is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with few cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods. There are some other ways as well, with more documentation on the internet.
 
== Wrapper script generation ==
'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

startexe="mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel"

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposePar &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable,
# in the header
echo $startexe
exec $startexe &&

# remove reconstructPar and the '&&' operator from the command above
# if you would like to reconstruct the case later
reconstructPar

</source>
|}
 

= OpenFOAM and ParaView on bwUniCluster=
ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/4.3.1</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
<pre>$ paraview</pre>
'''NOTICE:''' ParaView is a visualization software which requires X-server to run.

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.

----
[[Category:Engineering software]][[Category:bwUniCluster]]

BwUniCluster3.0/Software/OpenFoam

2015-10-26T16:05:25Z

A Saramet: /* Wrapper script generation */

= OpenFOAM =
[[File:OpenfoamLogo.png]]

Module: cae/openfoam/'version'
Target System: Red-Hat-Enterprise-Linux-Server-release-6.4-Santiago
Main Location: /opt/bwhpc/common
Priority: mandatory
License: GPL
Homepage: http://www.openfoam.org/

== Accessing and basic usage ==
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.

In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.

Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>

== Improving parallel run performance ==

To improve the concurrently solving process and decrease the error occurrence probability while running an OpenFOAM job in parallel (''on a singlenode''), some modifications have been introduced. Specifically, after the case decomposition is done, it is recommended to save the decomposed data directly on the nodes in the pre-allocated work-space and use it from there. When the calculations are over, the data is moved back to the case folder and reconstructed. It will improve the overall performance, considering that you allocated enough wall-time to decompose and rebuild your cases, as it moves theprocessor*folders to and out of the nodes local work-space. For such a procedure it is necessary to use the following commands for decomposition and reconstruction of the geometry domain:
<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>
instead of:
<pre>$ decomposPar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMesh -constant</pre>

= Building an OpenFOAM batch file for parallel processing =
== General information ==

To run any job in a parallel mode with OpenFOAM, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use. That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'' mode, letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between. There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results. Decomposition and segments building process is handled bydecomposeParutility. In "''system/decomposeParDict''" you may specify, how many "segments" you want your geometry domain to be divided in, and which decomposition method to use. The automatic one is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with few cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods. There are some other ways as well, with more documentation on the internet.

== Wrapper script generation ==

'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

startexe="mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel"

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposePar &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable,
#in the header
echo $startexe
exec $startexe &&

# remove reconstructPar if you would like to reconstruct the case later
reconstructPar

</source>
|}

= OpenFOAM and ParaView on bwUniCluster=

ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/4.3.1</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
<pre>$ paraview</pre>
'''NOTICE:''' ParaView is a visualization software which requires X-server to run.

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.
----
[[Category:Engineering software]][[Category:bwUniCluster]]

BwUniCluster3.0/Software/OpenFoam

2015-09-09T15:16:13Z

A Saramet: /* OpenFOAM and ParaView */

= OpenFOAM =
[[File:OpenfoamLogo.png]]

Module: cae/openfoam/'version'
Target System: Red-Hat-Enterprise-Linux-Server-release-6.4-Santiago
Main Location: /opt/bwhpc/common
Priority: mandatory
License: GPL
Homepage: http://www.openfoam.org/

== Accessing and basic usage ==
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.

In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.

Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>

== Improving parallel run performance ==

To improve the concurrently solving process and decrease the error occurrence probability while running an OpenFOAM job in parallel (''on a singlenode''), some modifications have been introduced. Specifically, after the case decomposition is done, it is recommended to save the decomposed data directly on the nodes in the pre-allocated work-space and use it from there. When the calculations are over, the data is moved back to the case folder and reconstructed. It will improve the overall performance, considering that you allocated enough wall-time to decompose and rebuild your cases, as it moves theprocessor*folders to and out of the nodes local work-space. For such a procedure it is necessary to use the following commands for decomposition and reconstruction of the geometry domain:
<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>
instead of:
<pre>$ decomposPar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMesh -constant</pre>

= Building an OpenFOAM batch file for parallel processing =
== General information ==

To run any job in a parallel mode with OpenFOAM, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use. That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'' mode, letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between. There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results. Decomposition and segments building process is handled bydecomposeParutility. In "''system/decomposeParDict''" you may specify, how many "segments" you want your geometry domain to be divided in, and which decomposition method to use. The automatic one is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with few cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods. There are some other ways as well, with more documentation on the internet.

== Wrapper script generation ==

'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

startexe="mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel"

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposePar &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable, in the header
echo $startexe
exec $startexe &&

# remove reconstructPar if you would like to reconstruct the case later
reconstructPar

</source>
|}

= OpenFOAM and ParaView on bwUniCluster=

ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/4.3.1</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
<pre>$ paraview</pre>
'''NOTICE:''' ParaView is a visualization software which requires X-server to run.

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.
----
[[Category:Engineering software]][[Category:bwUniCluster]]

BwUniCluster3.0/Software/OpenFoam

2015-09-09T15:15:59Z

A Saramet: /* OpenFOAM and ParaView */

= OpenFOAM =
[[File:OpenfoamLogo.png]]

Module: cae/openfoam/'version'
Target System: Red-Hat-Enterprise-Linux-Server-release-6.4-Santiago
Main Location: /opt/bwhpc/common
Priority: mandatory
License: GPL
Homepage: http://www.openfoam.org/

== Accessing and basic usage ==
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.

In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.

Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>

== Improving parallel run performance ==

To improve the concurrently solving process and decrease the error occurrence probability while running an OpenFOAM job in parallel (''on a singlenode''), some modifications have been introduced. Specifically, after the case decomposition is done, it is recommended to save the decomposed data directly on the nodes in the pre-allocated work-space and use it from there. When the calculations are over, the data is moved back to the case folder and reconstructed. It will improve the overall performance, considering that you allocated enough wall-time to decompose and rebuild your cases, as it moves theprocessor*folders to and out of the nodes local work-space. For such a procedure it is necessary to use the following commands for decomposition and reconstruction of the geometry domain:
<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>
instead of:
<pre>$ decomposPar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMesh -constant</pre>

= Building an OpenFOAM batch file for parallel processing =
== General information ==

To run any job in a parallel mode with OpenFOAM, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use. That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'' mode, letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between. There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results. Decomposition and segments building process is handled bydecomposeParutility. In "''system/decomposeParDict''" you may specify, how many "segments" you want your geometry domain to be divided in, and which decomposition method to use. The automatic one is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with few cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods. There are some other ways as well, with more documentation on the internet.

== Wrapper script generation ==

'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

startexe="mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel"

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposePar &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable, in the header
echo $startexe
exec $startexe &&

# remove reconstructPar if you would like to reconstruct the case later
reconstructPar

</source>
|}

= OpenFOAM and ParaView =

ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/4.3.1</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
<pre>$ paraview</pre>
'''NOTICE:''' ParaView is a visualization software which requires X-server to run.

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.
----
[[Category:Engineering software]][[Category:bwUniCluster]]

BwUniCluster3.0/Software/OpenFoam

2015-09-09T15:02:21Z

A Saramet: /* OpenFOAM and ParaView */

= OpenFOAM =
[[File:OpenfoamLogo.png]]

Module: cae/openfoam/'version'
Target System: Red-Hat-Enterprise-Linux-Server-release-6.4-Santiago
Main Location: /opt/bwhpc/common
Priority: mandatory
License: GPL
Homepage: http://www.openfoam.org/

== Accessing and basic usage ==
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.

In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.

Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>

== Improving parallel run performance ==

To improve the concurrently solving process and decrease the error occurrence probability while running an OpenFOAM job in parallel (''on a singlenode''), some modifications have been introduced. Specifically, after the case decomposition is done, it is recommended to save the decomposed data directly on the nodes in the pre-allocated work-space and use it from there. When the calculations are over, the data is moved back to the case folder and reconstructed. It will improve the overall performance, considering that you allocated enough wall-time to decompose and rebuild your cases, as it moves theprocessor*folders to and out of the nodes local work-space. For such a procedure it is necessary to use the following commands for decomposition and reconstruction of the geometry domain:
<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>
instead of:
<pre>$ decomposPar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMesh -constant</pre>

= Building an OpenFOAM batch file for parallel processing =
== General information ==

To run any job in a parallel mode with OpenFOAM, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use. That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'' mode, letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between. There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results. Decomposition and segments building process is handled bydecomposeParutility. In "''system/decomposeParDict''" you may specify, how many "segments" you want your geometry domain to be divided in, and which decomposition method to use. The automatic one is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with few cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods. There are some other ways as well, with more documentation on the internet.

== Wrapper script generation ==

'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

startexe="mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel"

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposePar &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable, in the header
echo $startexe
exec $startexe &&

# remove reconstructPar if you would like to reconstruct the case later
reconstructPar

</source>
|}

= OpenFOAM and ParaView =

ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened the specific ParaView module.

First load the ParaView module. For example:

<pre>$ module load cae/paraview/4.3.1</pre>

----
[[Category:Engineering software]][[Category:bwUniCluster]]

BwUniCluster3.0/Software/OpenFoam

2015-09-09T15:00:00Z

A Saramet: /* OpenFOAM and ParaView */

= OpenFOAM =
[[File:OpenfoamLogo.png]]

Module: cae/openfoam/'version'
Target System: Red-Hat-Enterprise-Linux-Server-release-6.4-Santiago
Main Location: /opt/bwhpc/common
Priority: mandatory
License: GPL
Homepage: http://www.openfoam.org/

== Accessing and basic usage ==
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.

In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.

Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>

== Improving parallel run performance ==

To improve the concurrently solving process and decrease the error occurrence probability while running an OpenFOAM job in parallel (''on a singlenode''), some modifications have been introduced. Specifically, after the case decomposition is done, it is recommended to save the decomposed data directly on the nodes in the pre-allocated work-space and use it from there. When the calculations are over, the data is moved back to the case folder and reconstructed. It will improve the overall performance, considering that you allocated enough wall-time to decompose and rebuild your cases, as it moves theprocessor*folders to and out of the nodes local work-space. For such a procedure it is necessary to use the following commands for decomposition and reconstruction of the geometry domain:
<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>
instead of:
<pre>$ decomposPar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMesh -constant</pre>

= Building an OpenFOAM batch file for parallel processing =
== General information ==

To run any job in a parallel mode with OpenFOAM, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use. That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'' mode, letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between. There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results. Decomposition and segments building process is handled bydecomposeParutility. In "''system/decomposeParDict''" you may specify, how many "segments" you want your geometry domain to be divided in, and which decomposition method to use. The automatic one is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with few cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods. There are some other ways as well, with more documentation on the internet.

== Wrapper script generation ==

'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

startexe="mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel"

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposePar &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable, in the header
echo $startexe
exec $startexe &&

# remove reconstructPar if you would like to reconstruct the case later
reconstructPar

</source>
|}

= OpenFOAM and ParaView =

ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened the specific ParaView module.

----
[[Category:Engineering software]][[Category:bwUniCluster]]

BwUniCluster3.0/Software/OpenFoam

2015-09-09T14:55:04Z

A Saramet:

= OpenFOAM =
[[File:OpenfoamLogo.png]]

Module: cae/openfoam/'version'
Target System: Red-Hat-Enterprise-Linux-Server-release-6.4-Santiago
Main Location: /opt/bwhpc/common
Priority: mandatory
License: GPL
Homepage: http://www.openfoam.org/

== Accessing and basic usage ==
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.

In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.

Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>

== Improving parallel run performance ==

To improve the concurrently solving process and decrease the error occurrence probability while running an OpenFOAM job in parallel (''on a singlenode''), some modifications have been introduced. Specifically, after the case decomposition is done, it is recommended to save the decomposed data directly on the nodes in the pre-allocated work-space and use it from there. When the calculations are over, the data is moved back to the case folder and reconstructed. It will improve the overall performance, considering that you allocated enough wall-time to decompose and rebuild your cases, as it moves theprocessor*folders to and out of the nodes local work-space. For such a procedure it is necessary to use the following commands for decomposition and reconstruction of the geometry domain:
<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>
instead of:
<pre>$ decomposPar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMesh -constant</pre>

= Building an OpenFOAM batch file for parallel processing =
== General information ==

To run any job in a parallel mode with OpenFOAM, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use. That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'' mode, letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between. There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results. Decomposition and segments building process is handled bydecomposeParutility. In "''system/decomposeParDict''" you may specify, how many "segments" you want your geometry domain to be divided in, and which decomposition method to use. The automatic one is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with few cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods. There are some other ways as well, with more documentation on the internet.

== Wrapper script generation ==

'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

startexe="mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel"

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposePar &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable, in the header
echo $startexe
exec $startexe &&

# remove reconstructPar if you would like to reconstruct the case later
reconstructPar

</source>
|}

= OpenFOAM and ParaView =
----
[[Category:Engineering software]][[Category:bwUniCluster]]

BwUniCluster3.0/Software/OpenFoam

2015-08-18T14:04:05Z

A Saramet: /* Wrapper script generation */

= OpenFOAM =
[[File:OpenfoamLogo.png]]

Module: cae/openfoam/'version'
Target System: Red-Hat-Enterprise-Linux-Server-release-6.4-Santiago
Main Location: /opt/bwhpc/common
Priority: mandatory
License: GPL
Homepage: http://www.openfoam.org/

== Accessing and basic usage ==
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.

In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.

Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>

== Improving parallel run performance ==

To improve the concurrently solving process and decrease the error occurrence probability while running an OpenFOAM job in parallel (''on a singlenode''), some modifications have been introduced. Specifically, after the case decomposition is done, it is recommended to save the decomposed data directly on the nodes in the pre-allocated work-space and use it from there. When the calculations are over, the data is moved back to the case folder and reconstructed. It will improve the overall performance, considering that you allocated enough wall-time to decompose and rebuild your cases, as it moves theprocessor*folders to and out of the nodes local work-space. For such a procedure it is necessary to use the following commands for decomposition and reconstruction of the geometry domain:
<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>
instead of:
<pre>$ decomposPar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMesh -constant</pre>

= Building an OpenFOAM batch file for parallel processing =
== General information ==

To run any job in a parallel mode with OpenFOAM, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use. That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'' mode, letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between. There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results. Decomposition and segments building process is handled bydecomposeParutility. In "''system/decomposeParDict''" you may specify, how many "segments" you want your geometry domain to be divided in, and which decomposition method to use. The automatic one is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with few cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods. There are some other ways as well, with more documentation on the internet.

== Wrapper script generation ==

'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

startexe="mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel"

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposePar &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable, in the header
echo $startexe
exec $startexe &&

# remove reconstructPar if you would like to reconstruct the case later
reconstructPar

</source>
|}
----
[[Category:Engineering software]][[Category:bwUniCluster]]

BwUniCluster3.0/Software/OpenFoam

2015-08-18T14:01:33Z

A Saramet: /* Wrapper script generation */

= OpenFOAM =
[[File:OpenfoamLogo.png]]

Module: cae/openfoam/'version'
Target System: Red-Hat-Enterprise-Linux-Server-release-6.4-Santiago
Main Location: /opt/bwhpc/common
Priority: mandatory
License: GPL
Homepage: http://www.openfoam.org/

== Accessing and basic usage ==
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.

In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.

Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>

== Improving parallel run performance ==

To improve the concurrently solving process and decrease the error occurrence probability while running an OpenFOAM job in parallel (''on a singlenode''), some modifications have been introduced. Specifically, after the case decomposition is done, it is recommended to save the decomposed data directly on the nodes in the pre-allocated work-space and use it from there. When the calculations are over, the data is moved back to the case folder and reconstructed. It will improve the overall performance, considering that you allocated enough wall-time to decompose and rebuild your cases, as it moves theprocessor*folders to and out of the nodes local work-space. For such a procedure it is necessary to use the following commands for decomposition and reconstruction of the geometry domain:
<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>
instead of:
<pre>$ decomposPar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMesh -constant</pre>

= Building an OpenFOAM batch file for parallel processing =
== General information ==

To run any job in a parallel mode with OpenFOAM, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use. That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'' mode, letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between. There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results. Decomposition and segments building process is handled bydecomposeParutility. In "''system/decomposeParDict''" you may specify, how many "segments" you want your geometry domain to be divided in, and which decomposition method to use. The automatic one is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with few cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods. There are some other ways as well, with more documentation on the internet.

== Wrapper script generation ==

'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

startexe="mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel"

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load cae/openfoam/2.4.0
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposePar &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable, in the header
echo $startexe
exec $startexe &&

# remove reconstructPar if you would like to reconstruct the case later
reconstructPar

</source>
|}
----
[[Category:Engineering software]][[Category:bwUniCluster]]

BwUniCluster3.0/Software/OpenFoam

2015-08-18T13:38:57Z

A Saramet: /* General information */

BwUniCluster3.0/Software/OpenFoam

2015-08-18T13:26:41Z

A Saramet: /* General information */

BwUniCluster3.0/Software/OpenFoam

2015-08-18T13:25:49Z

A Saramet: /* General information */

BwUniCluster3.0/Software/OpenFoam

2015-08-18T13:22:35Z

A Saramet: /* General information */

BwUniCluster3.0/Software/OpenFoam

2015-08-18T12:37:07Z

A Saramet: /* Building an OpenFOAM batch file for parallel processing */

BwUniCluster3.0/Software/OpenFoam

2015-08-18T11:52:51Z

A Saramet: /* Submitting an OpenFOAM batch job */

BwUniCluster3.0/Software/OpenFoam

2015-08-18T11:51:58Z

A Saramet: /* Improving parallel run performance */

BwUniCluster3.0/Software/OpenFoam

2015-08-18T11:50:04Z

A Saramet: /* Parallel Computing with OpenFOAM */

BwUniCluster3.0/Software/OpenFoam

2015-08-18T11:38:03Z

A Saramet: /* Parallel Computing with OpenFOAM */

BwUniCluster3.0/Software/OpenFoam

2015-08-18T11:32:50Z

A Saramet: /* Parallel Computing with OpenFOAM */

BwUniCluster3.0/Software/OpenFoam

2015-08-18T11:05:26Z

A Saramet: /* OpenFOAM */

File:OpenfoamLogo.png

2014-03-04T09:21:50Z

A Saramet: