Difference between revisions of "Development/Parallel Programming"

From bwHPC Wiki
Jump to: navigation, search
(Another restructure and a bit of content (nowait and MPI3))
(Open MPI)
Line 138: Line 138:
 
 
 
== Open MPI ==
 
== Open MPI ==
The [http://www.open-mpi.org|Open MPI] library is an open, flexible and nevertheless performant implementation of MPI-2 and MPI-3. Licensed under BSD, it is being actively developped by an open community of industry and research institutions.
+
The [http://www.open-mpi.org Open MPI] library is an open, flexible and nevertheless performant implementation of MPI-2 and MPI-3. Licensed under BSD, it is being actively developed by an open community of industry and research institutions.
The flexibility comes in handy: using the concept of a [http://www.open-mpi.org/faq/?category=tuning#mca-def|MCA] (aka a plugin) Open MPI supports many different network interconnects (Infinband, TCP, Cray, etc.) , on the other hand, a installation may be tailored to suite an installation, e.g. the network (Infiniband with specific settings), the main startup-mechanism, etc.
+
The flexibility comes in handy: using the concept of a [http://www.open-mpi.org/faq/?category=tuning#mca-def MCA] (aka a plugin) Open MPI supports many different network interconnects (Infinband, TCP, Cray, etc.) , on the other hand, a installation may be tailored to suite an installation, e.g. the network (Infiniband with specific settings), the main startup-mechanism, etc.
Furthermore, the [http://www.open-mpi.org/faq/|FAQ] offers hints on [http://www.open-mpi.org/faq/?category=tuning|performance tuning].
+
Furthermore, the [http://www.open-mpi.org/faq/ FAQ] offers hints on [http://www.open-mpi.org/faq/?category=tuning performance tuning].
   
 
=== Usage ===
 
=== Usage ===
Line 152: Line 152:
 
<tt>ompi_info --param ob1 all</tt> -- which may be increased for applications that are certain to benefit from a larger value upon startup.
 
<tt>ompi_info --param ob1 all</tt> -- which may be increased for applications that are certain to benefit from a larger value upon startup.
 
* Open MPI allows adapting MCA parameters on the command-line: parameters may be supplied, e.g. the above-mentioned parameter <tt>mpirun -np 128 --mca mpirun -np 16 --mca pml_ob1_free_list_num 128 ./mpi_stub</tt>.
 
* Open MPI allows adapting MCA parameters on the command-line: parameters may be supplied, e.g. the above-mentioned parameter <tt>mpirun -np 128 --mca mpirun -np 16 --mca pml_ob1_free_list_num 128 ./mpi_stub</tt>.
* Open MPI internally uses the tool [http://www.open-mpi.org/projects/hwloc/|hwloc] for node-local processor-information, as well as process- and memory-affinity. This tool also is a good tool to get information on the node's processor topology and Cache-information. This may be used to optimize and balance memory usage or for choosing a better ratio of MPI processes per node vs. OpenMP threads per core.
+
* Open MPI internally uses the tool [http://www.open-mpi.org/projects/hwloc/ hwloc] for node-local processor-information, as well as process- and memory-affinity. This tool also is a good tool to get information on the node's processor topology and Cache-information. This may be used to optimize and balance memory usage or for choosing a better ratio of MPI processes per node vs. OpenMP threads per core.
   
 
== Intel® MPI ==
 
== Intel® MPI ==

Revision as of 14:15, 4 February 2014

Navigation: bwHPC BPR

1 Introduction

This page will provide information regarding the supported parallel programming paradigms and specific hints on their usage. Please refer to the Modules Documentation how to setup your environment on bwUniCluster to load a specific software installation.

2 OpenMP

2.1 General Information

OpenMP is a mature [1] to allow easy, portable, and most importantly incremental node-level parallelisation of code. Being a thread-based approach, OpenMP is aimed at more fine-grained parallelism than MPI. Although there have been extensions to extend OpenMP for inter-node parallelisation, it is a node-level approach aimed to make best usage of a node's cores -- the section Hybrid Parallelisation will explain how to parallelise utilizing MPI plus a thread-based parallelization paradigm like OpenMP.

With regard to ease-of-use, OpenMP is ahead of any other common approach: the source-code is annotated using #pragma omp or !$omp statements, in C/C++ and Fortran respectively. Whenever the compiler encompasses a semantic block of code encapsulated in a parallel region, this block of code is transparently compiled into a function, which is passed to a so-called team-of-threads upon entering this semantic block. This fork-join model of execution eases a lot of the programmer's pain involved with Threads. Being a loop-centric approach, OpenMP is aimed at codes with long/time-consuming loops. A single combined directive pragma omp parallel for will tell the compiler to automatically parallelize the ensuing for-loop.

The following example is a bit more advanced in that even reductions of variables over multiple threads are easily parallelizable:

   for (int i=0, sum = 0.0; i < VECTOR_LEN; i++)
     norm2 += (v[i]*v[i]);

is parallelized by just adding a single line as in:

#  pragma omp parallel for reduction(+:norm2)
   for (int i=0, sum = 0.0; i < VECTOR_LEN; i++)
     norm2 += (v[i]*v[i]);

With VECTOR_LENGTH being large enough, this piece of code compiled with OpenMP will run in parallel, exhibiting very nice speedup. Compiled without, the code remains as is. Developpers may therefore incrementally parallelize their application based on the profile derived from performance analysis tools, starting with the most time-consuming loops. Using OpenMP's concise API, one may query the number of running threads, the number of processors, a time to calculate runtime, and even set parameters such as the number of threads to execute a parallel region.

The OpenMP-4.0 specification added support for the SIMD-directive to better utilize SIMD-vectorization, as well as integrating directives to offload computation to accelerators using the target directive: these are integrated into the Intel Compiler and are actively being worked on for the GNU compiler, some restrictions may apply.


2.2 OpenMP Best Practice Guide

The following silly example to calculate the squared Euklidian Norm shows some techniques:

#include <stdlib.h>
#include <stdio.h>
#include <omp.h>

#define VECTOR_LENGTH 5

int main (int argc, char * argv[])
{
    int len = VECTOR_LENGTH;
    int i;
    double * v;
    double norm2 = 0.0;
    double t1, tdiff;

    if (argc > 1)
        len = atoi (argv[1]);
    v = malloc (len * sizeof(double));

    t1 = omp_get_wtime();
    // Initialization already with (the same number of) threads
#pragma omp parallel for
    for (i=0; i < len; i++) {
        v[i] = i;
    }

    // Now aggregate the sum-of-squares by specifying a reduction
#pragma omp parallel for reduction(+:norm2)
    for(i=0; i < len; i++) {
        norm2 += (v[i]*v[i]);
    }
    tdiff = omp_get_wtime() - t1;

    printf ("norm2: %f Time:%f\n", norm2, tdiff);
    return 0;
}
  • Group independent parallel sections together: in the above example, You may combine those two sections into one larger parallel block. This will just once enter the parallel region (in the fork-join model) instead of twice. Especially in inner loops, this will considerably decrease overhead.
  • Compile with the Intel compiler's option -diag-enable sc-parallel3 to get the further warnings on thread-safety, performance, etc. The following code with loop-carried dependency will e.g. compile fine (aka without warning):
#pragma omp parallel for reduction(+:norm2)
    for(i=1; i < len-1; i++) {
        v[i] = v[i-1]+v[i+1];
    }

However the Intel compiler with -diag-enable sc-parallel3 will produce the following warning: warning #12246: variable "v" has loop carried data dependency that may lead to incorrect program execution in parallel mode; see (file:omp_norm2.c line:32)

  • Always specify default(none) on larger parallel regions in order to specifically set the visibility of variables to either shared or private.
  • Try to restructure code to allow for nowait: OpenMP defines synchronization points (implied barriers) at the end of worksharing constructs such as the pragma omp for directive. If the ensuing section of code does not depend on data being generated inside the parallel section, adding the nowait clause to the worksharing directive allows the compiler to eliminate this synchronization point. This reduces overhead, allows for better overlap and better utilization of the processor's ressources. This might imply however to restructure the code (move portions of independent code in between dependent works-sharing constructs).


2.3 Usage

OpenMP is supported by various compilers, here the usage for two main compilers GCC and Intel Suite are introduced. For both compilers, You first need to turn on OpenMP support by specifying a parameter on the compiler's command-line. In case You make function calls to OpenMP's API, You also need to include the header-file omp.h. OpenMP's API allows to query or set the number of threads, query the number of processors, get a wall-clock time to measure execution times, etc.

2.3.1 OpenMP with GNU Compiler Collection

Starting with version 4.2 the gcc compiler supports OpenMP-2.5. Since then the analysis capabilities of the GNU compiler have steadily improved. The installed compilers support OpenMP-3.1.

To use OpenMP with the gcc-compiler, pass -fopenmp as parameter.

2.3.2 OpenMP with Intel Compiler

The Intel Compiler's support for OpenMP is more advanced than gcc's -- especially in term of programmer support. To use OpenMP with the Intel compiler, pass -openmp as command-line parameter.

One may get very insightful information about OpenMP, when compiling with

  • Compiling with -openmp-report2 to get information, which loops were parallelized and a reason why not.
  • Compiling with -diag-enable sc-parallel3 to get errors and warnings about your sources weaknesses with regard to parallelization (see example below).


3 MPI

In this section, You will find information regarding the supported installations of the Message-Passing Interface libraries and their usage. Due to the Fortran interface ABI, all MPI-libraries are normally bound to a specific compiler-vendor and even the specific compiler version. Therefore, as listed in BwHPC_BPG_Compiler two compilers are supported on bwUniCluster: GCC and Intel Suite. As both compilers are continously improving, the communication libraries will be adopted in lock-step.

With a set of different implementations, there comes the problem of choice. These pages should inform the user of the communication libraries, what considerations should be done with regard to performance, maintainability and debugging -- in general tool support -- of the various implementations.

3.1 MPI Best Practice Guide

Specific performance considerations with regard to MPI (independent of the implementation):

  • No communication at all is best: Only communicate between processes if at all necessary. Consider that file-access is "communication" as well.
  • If communication is done with multiple processes, try to involve as many processes in just one call: MPI optimizes the communication pattern for so-called "collective communication" to take advantage of the underlying network (with regard to network topology, message sizes, queueing capabilities of the network interconnect, etc.). Therefore try to always think in collective communication, if a communication pattern involves a group of processes.
  • Try to group processes together: Function calls like MPI_Cart_create will come in handy for applications with cartesian domains but also general communicators derived from MPI_COMM_WORLD using MPI_Comm_splot() may benefit by MPI's knowing the underlying network topology. Use MPI3's MPI_Comm_split_type() with MPI_COMM_TYPE_SHARED for a sub-communicator with processes having access to the same shared memory region (aka the same node).
  • File-accesses to load / store data must be done collectively: Writing to storage, or even reading the initialization data -- all of which involves getting data from/to all MPI processes must be done collectively. MPI's Parallel IO implementation offers a rich API to read and distribute the data access -- in order to take advantage of parallel filesystems like Lustre. A many-fold performance improvement may be seen by writing data in large chunks in collective fashion -- and at the same time being nice to other users and applications.
  • Try to hide the communication by computation: Try to hide (some) of the cost of communication of Point-to-point communication by using non-blocking / immediate P2P-calls (MPI_Isend and MPI_Irecv et al, followed by MPI_Wait or MPI_Test et al). This may allow the MPI-implementation to initiate or even offload communication to the network interconnect and resume executing your application, while data is being transferred. MPI-3 adds non-blocking collectives, e.g. MPI_Ibcast() or MPI_Iallreduce(). For extra credit, explain the reasons of MPI_Ibarrier().
  • Every call to MPI may trigger an access to physical hardware -- limit it: When calling communication-related functions such as MPI_Test to check whether a specific communication has finished, the queue of the network adapter may need to be queried. This memory access or even physical hardware access to query the state will cost cycles. Therefore, the programmer should combine multiple requests with functions such as MPI_Waitall or MPI_Waitany() or their Test*-counterparts.
  • Make usage of derived datatypes: instead of manually copying data into temporary, even newly allocated memory, describe the data-layout to MPI -- and let the implementation, or even the network HCA's hardware do the data fetching.
  • Bind Your processes to sockets: Operating Systems are good in making best use of the ressources -- which sometimes involves moving tasks from one core to another, or even (though more unlikely since the OS' heuristics try to avoid it) to another socket, with the obvious effects (Caches are cold, every memory access to memory allocated on the previous socket "has to travel the bus"). This is particularly happening if You have multiple OpenMP parallel regions which are separated by IO -- and threads are sleeping -- the processes doing IO may wander to a different socket... Bind Your processes to at least the socket. All major MPIs support this binding (see below).
  • Do not use the C++ interface: First of all, it has been marked as deprecated in the MPI-3.0 standard, since it added little benefit to C++ programmers over the C-interface. Moreover, since MPI implementations are written in C, the interface adds another level of indirection and therefore a bit of overhead in terms of instructions and Cache misses.


3.2 Open MPI

The Open MPI library is an open, flexible and nevertheless performant implementation of MPI-2 and MPI-3. Licensed under BSD, it is being actively developed by an open community of industry and research institutions. The flexibility comes in handy: using the concept of a MCA (aka a plugin) Open MPI supports many different network interconnects (Infinband, TCP, Cray, etc.) , on the other hand, a installation may be tailored to suite an installation, e.g. the network (Infiniband with specific settings), the main startup-mechanism, etc. Furthermore, the FAQ offers hints on performance tuning.

3.2.1 Usage

Like other MPI implementations, after loading the module, Open MPI provides the compiler-wrappers mpicc, mpicxx and the various Fortran-representatives mpif77 for the C-, C++ and Fortran compilers respectively. Albeit their usage is not required, these wrappers are handy to not have to use the command-line options for header- or library directories, aka -I and -L, as well as the actual needed MPI-libraries itselve.

3.2.2 Further information

Open MPI also features a few specific functionalities that will help users and developpers, alike:

  • Open MPI's tool ompi_info allows seeing all of Open MPI's installed MCA components and their specific options.

Without any option the user gets a list of flags, the Open MPI installation was compiled for (version of compilers, specific configure-flags, e.g. debugging, or profiling options). Furthermore, using ompi_info --param all all one may see all of the MCA's options, e.g. that the default PML-MCA uses an initial free-list of 4 blocks (increased by 64 upon first encountering this limit): ompi_info --param ob1 all -- which may be increased for applications that are certain to benefit from a larger value upon startup.

  • Open MPI allows adapting MCA parameters on the command-line: parameters may be supplied, e.g. the above-mentioned parameter mpirun -np 128 --mca mpirun -np 16 --mca pml_ob1_free_list_num 128 ./mpi_stub.
  • Open MPI internally uses the tool hwloc for node-local processor-information, as well as process- and memory-affinity. This tool also is a good tool to get information on the node's processor topology and Cache-information. This may be used to optimize and balance memory usage or for choosing a better ratio of MPI processes per node vs. OpenMP threads per core.

3.3 Intel® MPI

under construction.


4 Hybrid Parallelization

under construction.