Development/General compiler usage - Revision history

H Winkhardt: UC3 Betriebsmodell MPI

2025-04-14T07:46:55Z

UC3 Betriebsmodell MPI

← Older revision		Revision as of 09:46, 14 April 2025
Line 28:		Line 28:

	Since Fortran (and very old C++) requires compiling and linking libraries with the very same compiler, many libraries, first-and-foremost the MPI libraries need to be provided for specific versions of a compiler.		Since Fortran (and very old C++) requires compiling and linking libraries with the very same compiler, many libraries, first-and-foremost the MPI libraries need to be provided for specific versions of a compiler.
	On ~~[[BwUniCluster_2.0]]~~, these provided libraries will only be visible to <kbd>module avail</kbd>, once a compiler is loaded.		On some clusters, these provided libraries will only be visible to <kbd>module avail</kbd>, once a compiler is loaded.
	Hence, check out loading		Hence, check out loading
	<pre>		<pre>

R Keller at 10:54, 23 January 2025

2025-01-23T10:54:31Z

← Older revision		Revision as of 12:54, 23 January 2025
Line 178:		Line 178:
	return EXIT_SUCCESS; // End program by returning 0 (No Error)		return EXIT_SUCCESS; // End program by returning 0 (No Error)
	}</source>		}</source>
	After loading the Intel Compiler module ''module load compiler/intel/'', the source may be compiled and linked with the single command		After loading the Intel Compiler module using <kbd>module load compiler/intel/</kbd>, the source may be compiled and linked with the single command
	<pre>$ icc hello.c -o hello</pre>		<pre>$ icc hello.c -o hello</pre>
	to produce an executable named ''hello''. This may be executed using the command ''./hello''.		to produce an executable named <kbd>hello</kbd>. This may be executed using the command <kbd>./hello</kbd>.


	This process can be divided into two steps:		This process can be divided into two steps:
	<pre>		<pre>
	$ icc -c hello.c # Compile .c File into object file (ending in .o)		$ icc -c hello.c # Compile .c File into object file (ending in .o)
	$ icc hello.o -o hello # Link the object file with the system libc library)		$ icc hello.o -o hello # Link the object file with the system libc library)
	</pre>		</pre>
	When using libraries you must sometimes specify the directories where the		When using libraries you must sometimes specify the directories where the
Line 279:		Line 279:

	[[File:static_code_analysis.png\|right\|border\|513px\|Copyright: HS Esslingen)]]		[[File:static_code_analysis.png\|right\|border\|513px\|Copyright: HS Esslingen)]]
	Another powerful feature available in GNU- and LLVM-based compilers is ''static code analysis''', otherwise only available in Commercial tools, like [https://www.synopsys.com/software-integrity/security-testing/static-analysis-sast.html Coverity].		Another powerful feature available in GNU- and LLVM-based compilers is '''static code analysis''', otherwise only available in Commercial tools, like [https://www.synopsys.com/software-integrity/security-testing/static-analysis-sast.html Coverity].
	Static code analysis evaluates '''each''' and '''every''' code path, making assumptions on input values and branches taken, detecting corner cases which might lead to real errors -- without having to actually execute this code path.		Static code analysis evaluates '''each''' and '''every''' code path, making assumptions on input values and branches taken, detecting corner cases which might lead to real errors -- without having to actually execute this code path.

R Keller at 10:51, 23 January 2025

2025-01-23T10:51:35Z

← Older revision		Revision as of 12:51, 23 January 2025
Line 39:		Line 39:
	to see the available MPI modules.		to see the available MPI modules.

	All vendors whether it's Intel, GNU, LLVM or the Nvidia toolkit (see toolkit) have compilers for different programming languages which will be available		All vendors whether it's Intel, GNU, LLVM or the Nvidia toolkit (see module group toolkit) have compilers for different programming languages which will be available
	only after loading the module.		only after loading the module.

Line 128:		Line 128:
	== MPI compiler and Underlying Compilers ==		== MPI compiler and Underlying Compilers ==

	MPI implementations such as MPIch, Intel-MPI (derived from MPIch) or Open MPI provide compiler wrappers, easing the usage of MPI by providing the Include-Directory <kbd>-I</kbd> and required libraries as well as the ~~MPI implementations~~ library ~~directorie~~ <kbd>-L</kbd> for linking.		MPI implementations such as MPIch, Intel-MPI (derived from MPIch) or Open MPI provide compiler wrappers, easing the usage of MPI by providing the Include-Directory <kbd>-I</kbd> and required libraries as well as the implementation's library directory flag <kbd>-L</kbd> for linking.
	The following table lists available MPI compiler commands and the underlying compilers, compiler families, languages, and application binary interfaces (ABIs) that they support.		The following table lists available MPI compiler commands and the underlying compilers, compiler families, languages, and application binary interfaces (ABIs) that they support.

Line 178:		Line 178:
	return EXIT_SUCCESS; // End program by returning 0 (No Error)		return EXIT_SUCCESS; // End program by returning 0 (No Error)
	}</source>		}</source>
	It may be compiled and linked with the single command		After loading the Intel Compiler module ''module load compiler/intel/'', the source may be compiled and linked with the single command
	<pre>$ icc hello.c -o hello</pre>		<pre>$ icc hello.c -o hello</pre>
	to produce an executable named ''hello''.		to produce an executable named ''hello''. This may be executed using the command ''./hello''.


	This process can be divided into two steps:		This process can be divided into two steps:
	<pre>		<pre>
	$ icc -c hello.c		$ icc -c hello.c # Compile .c File into object file (ending in .o)
	$ icc hello.o -o hello		$ icc hello.o -o hello # Link the object file with the system libc library)
	</pre>		</pre>
	When using libraries you must sometimes specify where the		When using libraries you must sometimes specify the directories where the
	* include files ~~are~~ (option <kbd>-I</kbd>) and where the		* include files (option <kbd>-I</kbd>) and where the
	* library files ~~are~~ (option <kbd>-L</kbd>).		* library files (option <kbd>-L</kbd>) are located.
	In addition you have to tell the compiler which		In addition you have to tell the compiler which
	* library you want to ~~use~~ (option <kbd>-l</kbd>).		* library you want to link to (option <kbd>-l</kbd>).
	For example after ~~loading the~~ module numlib/fftw you can compile code for fftw using		For example after <kbd>module load numlib/fftw</kbd> you can compile code for fftw using
	<pre>		<pre>
	$ icc -c hello.c -I$FFTW_INC_DIR		$ icc -c hello.c -I$FFTW_INC_DIR
Line 235:		Line 235:

	Please note, that the optimization level <kbd>-O2</kbd> produces code for a general instruction set.		Please note, that the optimization level <kbd>-O2</kbd> produces code for a general instruction set.
	If you want to set the instruction set available, and take advantage of AVX2 or AVX512f, you have to		If you want to set the instruction set available, and take advantage of AVX2 or AVX512f/AVX10.2, you have to
	either add the machine-dependent <kbd>-mavx512f</kbd> or set the specific architecture of your		either add the machine-dependent <kbd>-mavx512f</kbd> or set the specific architecture of your
	target processor.		target processor.
Line 271:		Line 271:
	E.g. <kbd>-Wextra</kbd> turns on several other warnings, which will in the above example show that neither <kbd>argc</kbd> nor <kbd>argv</kbd> have been used inside of <kbd>main</kbd>.		E.g. <kbd>-Wextra</kbd> turns on several other warnings, which will in the above example show that neither <kbd>argc</kbd> nor <kbd>argv</kbd> have been used inside of <kbd>main</kbd>.

	For LLVM's <kbd>clang</kbd> the flag <kbd>-Weverything</kbd> turns on all available warnings, albeit leading to many warnings on larger projects.		For LLVM's <kbd>clang</kbd> the flag <kbd>-Weverything</kbd> turns on all available warnings, albeit leading to many warnings (even false positives) on larger projects.
	However, the fix-it hints are very helpful as well.		However, the fix-it hints are very helpful as well.

Line 279:		Line 279:

	[[File:static_code_analysis.png\|right\|border\|513px\|Copyright: HS Esslingen)]]		[[File:static_code_analysis.png\|right\|border\|513px\|Copyright: HS Esslingen)]]
	Another powerful feature available in GNU- and LLVM-compilers is ''static code analysis''', otherwise only available in Commercial tools, like [https://www.synopsys.com/software-integrity/security-testing/static-analysis-sast.html Coverity].		Another powerful feature available in GNU- and LLVM-based compilers is ''static code analysis''', otherwise only available in Commercial tools, like [https://www.synopsys.com/software-integrity/security-testing/static-analysis-sast.html Coverity].
	Static code analysis evaluates '''each''' and '''every''' code path, making assumptions on input values and branches taken, detecting corner cases which might lead to real errors -- without having to actually execute this code path.		Static code analysis evaluates '''each''' and '''every''' code path, making assumptions on input values and branches taken, detecting corner cases which might lead to real errors -- without having to actually execute this code path.

Line 290:		Line 290:

	This produces warnings on <kbd>stdout</kbd>, but more importantly scan reports in directory <kbd>/scratch/scan-build-XXX</kbd>, where XXX is date and time of the build.		This produces warnings on <kbd>stdout</kbd>, but more importantly scan reports in directory <kbd>/scratch/scan-build-XXX</kbd>, where XXX is date and time of the build.
	For example the output of Open MPI includes real issues of missed memory releases in error code paths:		For example the output of Open MPI includes real issues of missed memory releases in error code paths -- as shown in the following picture.

R Keller: Update the versions to match the current (bwUniCluster 2.0) expectations

2025-01-23T10:20:18Z

Update the versions to match the current (bwUniCluster 2.0) expectations

← Older revision		Revision as of 12:20, 23 January 2025
Line 24:		Line 24:
	Modules and loading of modules is described in general in [[Environment_Modules\|here for traditional Environment Modules]] and about the Lmod implementation of Environment modules in [[Software_Modules_Lmod\|here for Lmod]].		Modules and loading of modules is described in general in [[Environment_Modules\|here for traditional Environment Modules]] and about the Lmod implementation of Environment modules in [[Software_Modules_Lmod\|here for Lmod]].

	~~However, modules~~ need to be mentioned, since on any system there's a pre-installed set of compilers (for C, C++ and usually Fortran), ~~which~~ ~~are~~ provided by the Linux distribution ~~-- the so-called~~ system ~~compilers. Which~~ however may lack certain options for optimization, for warnings or other features. On RedHat Enterprise Linux this is GNU compiler v8.3.1.		Modules need to be mentioned, since on any system there's a pre-installed set of compilers (for C, C++ and usually Fortran) -- the so-called system compilers -- provided by the Linux distribution. The system compiler however may lack certain options for optimization, are typically optimizing for older and more architectures (think SSE vs. AVX2 and AVX-512/AVX10.2). Also they may lack useful warnings or other features. On RedHat Enterprise Linux 9.4 this is GNU compiler v11.4.1.
	Be advised to check out the newer compilers available as modules.		Be advised to check out the newer compilers available as modules.

Line 33:		Line 33:
	$ module avail compiler/intel		$ module avail compiler/intel
	...		...
	$ module load compiler/intel/~~2021~~.4.0		$ module load compiler/intel/2023.1.0
	...		...
	$ module avail		$ module avail
Line 39:		Line 39:
	to see the available MPI modules.		to see the available MPI modules.

	All Intel, ~~GCC~~ ~~and~~ ~~PGI~~ have compilers for different programming languages which will be available		All vendors whether it's Intel, GNU, LLVM or the Nvidia toolkit (see toolkit) have compilers for different programming languages which will be available
	after the module ~~is loaded~~.		only after loading the module.


Line 50:		Line 50:
	<u>Example</u>		<u>Example</u>
	<pre>		<pre>
	$ module purge # unload all modules		$ module purge # unload all modules
	$ module list # ~~control~~		$ module list # check which modules are loaded, we expect none
	No Modulefiles Currently Loaded.		No Modulefiles Currently Loaded.
	$ gcc --version # see version of default Linux GNU compiler		$ gcc --version # see version of default Linux GNU compiler
	gcc (GCC) 8.3.1 ~~20191121~~ (Red Hat 8.3.1-5)		gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-18)
	[...]		[...]
	$ module load compiler/gnu # load default GNU compiler module		$ module load compiler/gnu # load default GNU compiler module
	$ module list # ~~control~~		$ module list # check which modules are loaded, we expect the default GNU compiler
	Currently Loaded Modulefiles:		Currently Loaded Modulefiles:
	1) compiler/gnu/10.~~2(default)~~		1) compiler/gnu/13.3
	$ gcc --version # now, check the current (loaded) ~~module~~		$ gcc --version # now, check the current (loaded) version of the GNU C compiler
	gcc (GCC) 10.2.0		gcc (GCC) 13.3.0
	[...]		[...]
	</pre>		</pre>

R Keller: Order of links switched, since the latter is much more thorough

2025-01-23T09:28:21Z

Order of links switched, since the latter is much more thorough

← Older revision		Revision as of 11:28, 23 January 2025
Line 22:		Line 22:
	= Loading compilers as modules =		= Loading compilers as modules =

	Modules and loading of modules is described [[~~Software_Modules_Lmod~~\|here for ~~Lmod~~]] and [[~~Environment_Modules~~\|here for ~~traditional Environment Modules~~]].		Modules and loading of modules is described in general in [[Environment_Modules\|here for traditional Environment Modules]] and about the Lmod implementation of Environment modules in [[Software_Modules_Lmod\|here for Lmod]].

	However, modules need to be mentioned, since on any system there's a pre-installed set of compilers (for C, C++ and usually Fortran), which are provided by the Linux distribution -- the so-called system compilers. Which however may lack certain options for optimization, for warnings or other features. On RedHat Enterprise Linux this is GNU compiler v8.3.1.		However, modules need to be mentioned, since on any system there's a pre-installed set of compilers (for C, C++ and usually Fortran), which are provided by the Linux distribution -- the so-called system compilers. Which however may lack certain options for optimization, for warnings or other features. On RedHat Enterprise Linux this is GNU compiler v8.3.1.

S Richling: /* Debugger */

2022-12-08T23:14:18Z

Debugger

← Older revision		Revision as of 01:14, 9 December 2022
Line 206:		Line 206:

	If the problem can't be solved this way you can inspect what exactly your program		If the problem can't be solved this way you can inspect what exactly your program
	does ~~[[BwHPC_BPG_Debugger\|~~using a debugger]].		does using a debugger, e.g. [[Development/GDB\|GDB]].

	<font color=green>To use the debugger properly with your program you have to compile it with debug information (option <kbd>-g</kbd>)</font>:		<font color=green>To use the debugger properly with your program you have to compile it with debug information (option <kbd>-g</kbd>)</font>:
Line 215:		Line 215:
	debugging purposes to find bugs.		debugging purposes to find bugs.
	It may slow down execution and enlarges the binary due to debugging symbols.		It may slow down execution and enlarges the binary due to debugging symbols.


	== Optimization ==		== Optimization ==

S Richling: /* Description */

2022-12-08T22:20:38Z

Description

← Older revision		Revision as of 00:20, 9 December 2022
Line 15:		Line 15:
	Basically, compilers translate human-readable source code (e.g. C++ interpreted as adhering to ISO/IEC 14882:2014, encoded in UTF-8 text) into binary byte code (e.g. x86-64 with Linux ABI in ELF-format).		Basically, compilers translate human-readable source code (e.g. C++ interpreted as adhering to ISO/IEC 14882:2014, encoded in UTF-8 text) into binary byte code (e.g. x86-64 with Linux ABI in ELF-format).
	Compilers are complex software and have become very powerful in the last decades, to '''guide''' you as a programmer writing better, more portable, more performant programs. Use the compiler as a tool -- and best use multiple compilers on the same source code for best results.		Compilers are complex software and have become very powerful in the last decades, to '''guide''' you as a programmer writing better, more portable, more performant programs. Use the compiler as a tool -- and best use multiple compilers on the same source code for best results.
	The basic operations and hints can be performed with the same or similar commands on all available compilers. For advanced usage such as optimization and profiling you should consult the best practice guide of the compiler you intend to use ([[~~BwHPC_BPG_Compiler#~~GCC\|GCC]], [[~~BwHPC_BPG_Compiler#Intel Suite~~\|Intel Suite]]).		The basic operations and hints can be performed with the same or similar commands on all available compilers. For advanced usage such as optimization and profiling you should consult the best practice guide of the compiler you intend to use ([[Development/GCC\|GCC]], [[Development/Intel_Compiler\|Intel Suite]]).

	More information about the MPI versions of the GNU and Intel Compilers is available here:		More information about the MPI versions of the GNU and Intel Compilers is available here:

S Richling at 22:19, 8 December 2022

2022-12-08T22:19:34Z

← Older revision		Revision as of 00:19, 9 December 2022
Line 143:		Line 143:
	\| mpifc \|\| gfortran \|\| Fortran77/Fortran 95 \|\| 32/64 bit		\| mpifc \|\| gfortran \|\| Fortran77/Fortran 95 \|\| 32/64 bit
	\|-		\|-
	\| colspan=4 style="background-color:#DCDCDC;" \| [[GCC\|GNU Compiler]] Versions 3 and higher		\| colspan=4 style="background-color:#DCDCDC;" \| [[Development/GCC\|GNU Compiler]] Versions 3 and higher
	\|-		\|-
	\| mpigcc \|\| gcc \|\| C \|\| 32/64 bit		\| mpigcc \|\| gcc \|\| C \|\| 32/64 bit
Line 227:		Line 227:

	All compilers offer a multitude of optimization options,		All compilers offer a multitude of optimization options,
	one may check the complete list of options with short explanation on [[GCC\|GCC]], [[LLVM\|LLVM]] and		one may check the complete list of options with short explanation on [[Development/GCC\|GCC]], [[LLVM\|LLVM]] and
	[[Development/Intel_Compiler\|Intel Suite]] using option '''-v''' '''--help''':		[[Development/Intel_Compiler\|Intel Suite]] using option '''-v''' '''--help''':
	<pre>		<pre>

S Richling at 22:16, 8 December 2022

2022-12-08T22:16:15Z

← Older revision		Revision as of 00:16, 9 December 2022
Line 153:		Line 153:
	\| mpif90 \|\| gfortran \|\| Fortran 95 \|\| 32/64 bit		\| mpif90 \|\| gfortran \|\| Fortran 95 \|\| 32/64 bit
	\|-		\|-
	\| colspan=4 style="background-color:#DCDCDC;" \| [[Intel_Compiler\|Intel Fortran, C++ Compilers]] Versions 13.1 through 14.0 and Higher		\| colspan=4 style="background-color:#DCDCDC;" \| [[Development/Intel_Compiler\|Intel Fortran, C++ Compilers]] Versions 13.1 through 14.0 and Higher
	\|-		\|-
	\| mpiicc \|\| icc \|\| C \|\| 32/64 bit		\| mpiicc \|\| icc \|\| C \|\| 32/64 bit
Line 228:		Line 228:
	All compilers offer a multitude of optimization options,		All compilers offer a multitude of optimization options,
	one may check the complete list of options with short explanation on [[GCC\|GCC]], [[LLVM\|LLVM]] and		one may check the complete list of options with short explanation on [[GCC\|GCC]], [[LLVM\|LLVM]] and
	[[Intel_Compiler\|Intel Suite]] using option '''-v''' '''--help''':		[[Development/Intel_Compiler\|Intel Suite]] using option '''-v''' '''--help''':
	<pre>		<pre>
	$ icc -v --help \| less		$ icc -v --help \| less

S Richling: /* Compilers (no MPI) */

2022-12-08T22:14:37Z

Compilers (no MPI)

← Older revision		Revision as of 00:14, 9 December 2022
Line 95:		Line 95:
	\| ifx		\| ifx
	\|-		\|-
	\| style="vertical-align:top;" rowspan="3" \| <font color=green><big>GCC</big></font><br><br> [[GCC\|• Best Practice Guides on GNU Compiler Software]]		\| style="vertical-align:top;" rowspan="3" \| <font color=green><big>GCC</big></font><br><br> [[Development/GCC\|• Best Practice Guides on GNU Compiler Software]]
	\| C		\| C
	\| gcc		\| gcc