bwHPC Wiki - User contributions [en]

Development/GDB

2016-02-24T08:49:12Z

R Doros: /* Disassembling */ Loop disassembling

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| devel/gdb
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]]
|-
| License
| [http://www.gnu.org/licenses/ GPL]
|-
|Citing
| n/a
|-
| Links
| [http://www.gnu.org/software/gdb/ Homepage] | [http://www.gnu.org/software/gdb/documentation/ Documentation] | [https://sourceware.org/gdb/wiki/ Wiki] | [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists]
|-
| Graphical Interface
| No
|-
| Included modules
| icc | icpc | ifort | idb
|}
 
= Introduction =
The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/devel/gdb
|width=99%
|height=120
}}
On the command line interface of any bwHPC cluster you'll get a list of available versions
by using the command ''''module avail devel/gdb''''.
<pre>
$ : bwUniCluster
$ module avail devel/gdb
------------------------ /opt/bwhpc/common/modulefiles -------------------------
devel/gdb/7.7</pre>

Attention! 
The default Linux GDB may be a different version.
Check the default version by: ''''gdb --version'''' without loaded GDB-module.
<pre>
$ module clear # clear all loaded modules
Are you sure you want to clear all loaded modules!? [n] y
$ gdb --version # this is the default version
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-83.el6)
[...]
$ module avail devel/gdb # available GDB-module
------------------------ /opt/bwhpc/common/modulefiles -------------------------
devel/gdb/7.7
</pre>
''It is not recommended (or impossible) the debug MPI-based binaries with GDB. You can't attach to MPI'ed jobs, too.''
 
 

= Loading =
It is possible to choose between several versions of GDB. By default both programs are installed. You can check the version which is currently used with:
<pre>$ gdb --version</pre>
<pre>$ gcc --version</pre>
 
To use a different version of GCC or GDB, you have to load it through the module system. For example you need GDB 7.7 or higher for GCC version 4.8 and above. In this case load the devel/gdb module.
 
How to load the desired version (e.g. GDB version 7.7):
<pre>$ module load devel/gdb/7.7</pre>
 
If you want to use IDB load the Intel compiler module:
<pre>$ module load compiler/intel</pre>
 
= Documentation =
For '''online documentation''' see the links section in the summary table at the top of this page. For '''local documentation''' consult the man page.
<pre>$ man gdb</pre>
or
<pre>$ man idb</pre>
 
= Basic commands =
The code you want to debug should be compiled with the -g option. If the optimization flag is not set, the GCC will still do some basic optimization. Therefore, it is recommended to turn off the optmization explicitly with the -O0 parameter for debugging. To start a debug session for a program execute GDB with the program path as parameter:
<pre>$ gdb ./example</pre>

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

{| width=600px class="wikitable"
|-
! Command
! Description
|-
| help ''cmd''
| Show help for command ''cmd''.
|-
| break ''func''
| Set a breakpoint at function ''func''.
|-
| run
| Start program.
|-
| next
| Go to next program line. Do not enter functions.
|-
| step
| Go to next program line. Enter functions.
|-
| list
| Show the surrounding source code of the currently processed line.
|-
| print ''expr''
| Print the value of the expression ''expr''.
|-
| display ''expr''
| Display the value of the expression ''expr'' every time the program stops.
|-
| watch ''expr''
| Stop when value of the expression ''expr'' changes.
|-
| continue
| Continue execution until a breakpoint or a watchpoint appears.
|-
| backtrace
| Print a list of functions that are currently active.
|-
| quit
| Exit GDB.
|}

'''Example:'''
We debug the following program called bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>

int global = 0;

void begin() {
global = 1;
}

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}

void end() {
global = 2;
}

int main() {
begin();
loop();
end();

return 0;
}
</source>

'''Sample GDB session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27 loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
(gdb) next
27 loop();
(gdb) step
loop () at bug.c:13
13 for(i = 0; i < 8; i++)
(gdb) next
15 k = i/2*2;
(gdb) next
16 v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16 v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit
</pre>
 

= Core dumps =
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
<pre>$ ulimit -c unlimited</pre>
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
<pre>$ gdb ./ex core.xxx</pre>
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.
 
 

= Multithreaded debugging =
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step.
The following commands are useful for multithreaded debugging:

{| width=600px class="wikitable"
|-
! Command
! Description
|-
| info threads
| Shows the status of all existing threads.
|-
| thread ''num''
| Switches to the thread with the number ''num''
|}

'''Example:'''
We debug the following program called thread_bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>
#include <pthread.h>

pthread_t thread;

void* thread3 (void* d)
{
int w[2];
int c, l;

for(c = 0; c < 8; c++) {
l = c/2*2; /* should have been l = c/(2*2); */
w[l] = c;
}

return NULL;
}

void* thread2 (void* d)
{
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
sleep(4);
k = i/(2*2); /* should have been k = i/(2*2); */
v[k] = i;
}

return NULL;
}

int main (){

pthread_create (&thread, NULL, thread2, NULL);
pthread_create (&thread, NULL, thread3, NULL);

//Thread 1
int count1 = 0;

while(count1 < 4000){
printf("Thread 1: %d\n", count1++);
}

pthread_join(thread, NULL);
return 0;
}
</source>

'''Sample GDB thread session:'''
<pre>
$ gcc -g thread_bug.c -o thread_bug -lpthread
$ gdb ./thread_bug
[...]
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break thread3
Breakpoint 1 at 0x40060c: file thread_bug.c, line 11.
(gdb) break thread2
Breakpoint 2 at 0x400650: file thread_bug.c, line 24.
(gdb) break main
Breakpoint 3 at 0x40069e: file thread_bug.c, line 35.
(gdb) run
Starting program: /tank/home/doros/.t/thread_bug
[Thread debugging using libthread_db enabled]

Breakpoint 3, main () at thread_bug.c:35
35 pthread_create (&thread, NULL, thread2, NULL);
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64
(gdb) info threads
* 1 Thread 0x7ffff7fe5700 (LWP 28260) main () at thread_bug.c:35
(gdb) next
[New Thread 0x7ffff7fe3700 (LWP 28303)]
36 pthread_create (&thread, NULL, thread3, NULL);
(gdb) info threads
2 Thread 0x7ffff7fe3700 (LWP 28303) thread2 (d=0x0) at thread_bug.c:24
* 1 Thread 0x7ffff7fe5700 (LWP 28260) main () at thread_bug.c:36
(gdb) next
[Switching to Thread 0x7ffff7fe3700 (LWP 28303)]

Breakpoint 2, thread2 (d=0x0) at thread_bug.c:24
24 for(i = 0; i < 8; i++) {
(gdb) next
25 sleep(4);
(gdb) next
[New Thread 0x7ffff77e2700 (LWP 28344)]
[Switching to Thread 0x7ffff77e2700 (LWP 28344)]

Breakpoint 1, thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) info threads
* 3 Thread 0x7ffff77e2700 (LWP 28344) thread3 (d=0x0) at thread_bug.c:11
2 Thread 0x7ffff7fe3700 (LWP 28303) 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 28260) 0x000000362f8725db in _IO_new_file_overflow () from /lib64/libc.so.6
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff7fe3700 (LWP 28303))]#0 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
(gdb) next
Single stepping until exit from function nanosleep,
which has no line number information.
[Switching to Thread 0x7ffff77e2700 (LWP 28344)]

Breakpoint 1, thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff7fe3700 (LWP 28303))]#0 0x000000362f8acce9 in nanosleep () from /lib64/libc.so.6
(gdb) next
Single stepping until exit from function nanosleep,
which has no line number information.
0x000000362f8acb50 in sleep () from /lib64/libc.so.6
(gdb) info threads
3 Thread 0x7ffff77e2700 (LWP 28344) thread3 (d=0x0) at thread_bug.c:11
* 2 Thread 0x7ffff7fe3700 (LWP 28303) 0x000000362f8acb50 in sleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 28260) 0x000000362f8476f0 in vfprintf () from /lib64/libc.so.6
(gdb) thread 3
[Switching to thread 3 (Thread 0x7ffff77e2700 (LWP 28344))]#0 thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) next
12 l = c/2*2; /* should have been l = c/(2*2); */
(gdb) watch (k >= 2)
No symbol "k" in current context.
(gdb) watch (l >= 2)
Hardware watchpoint 4: (l >= 2)
(gdb) continue
Continuing.
Thread 1: 0
Thread 1: 1
Thread 1: 2
Thread 1: 3
Thread 1: 4
[...]
Hardware watchpoint 4: (l >= 2)

Old value = 0
New value = 1
thread3 (d=0x0) at thread_bug.c:13
13 w[l] = c;
(gdb) print l
$1 = 2
(gdb) print c
$2 = 2
(gdb) quit
</pre>
= Disassembling =
{| width=600px class="wikitable"
|-
! Command
! Description
|-
| info functions
| Shows names and data types of all defined functions.
|-
| info line "function"
| Map source lines to memory adresses (and back).
|-
| disassemble ''function''
| Disassembles "function" (or a function fragment).
|}
'''Sample GDB disassembling session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
[...]
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bug...done.
(gdb) info functions
All defined functions:

File bug.c:
void begin();
void end();
void loop();
int main();

Non-debugging symbols:
0x0000000000400370 _init
0x00000000004003a0 __libc_start_main@plt
0x00000000004003b0 __gmon_start__@plt
0x00000000004003c0 _start
0x00000000004003f0 deregister_tm_clones
0x0000000000400430 register_tm_clones
0x0000000000400470 __do_global_dtors_aux
0x0000000000400490 frame_dummy
0x0000000000400540 __libc_csu_init
0x00000000004005b0 __libc_csu_fini
0x00000000004005b4 _fini
</pre>
'''Sample GDB disassembling session:'''
<pre>
(gdb) disassemble main
Dump of assembler code for function main:
0x000000000040050f <+0>: push %rbp
0x0000000000400510 <+1>: mov %rsp,%rbp
0x0000000000400513 <+4>: mov $0x0,%eax
0x0000000000400518 <+9>: callq 0x4004b6 <begin>
0x000000000040051d <+14>: mov $0x0,%eax
0x0000000000400522 <+19>: callq 0x4004c7 <loop>
0x0000000000400527 <+24>: mov $0x0,%eax
0x000000000040052c <+29>: callq 0x4004fe <end>
0x0000000000400531 <+34>: mov $0x0,%eax
0x0000000000400536 <+39>: pop %rbp
0x0000000000400537 <+40>: retq
End of assembler dump.
</pre>
'''Sample GDB disassembling session:'''
<pre>
(gdb) disassemble /m main
Dump of assembler code for function main:
23 int main() {
0x000000000040050f <+0>: push %rbp
0x0000000000400510 <+1>: mov %rsp,%rbp

24 begin();
0x0000000000400513 <+4>: mov $0x0,%eax
0x0000000000400518 <+9>: callq 0x4004b6 <begin>

25 loop();
0x000000000040051d <+14>: mov $0x0,%eax
0x0000000000400522 <+19>: callq 0x4004c7 <loop>

26 end();
0x0000000000400527 <+24>: mov $0x0,%eax
0x000000000040052c <+29>: callq 0x4004fe <end>

27
28 return 0;
0x0000000000400531 <+34>: mov $0x0,%eax

29 }
0x0000000000400536 <+39>: pop %rbp
0x0000000000400537 <+40>: retq

End of assembler dump.
</pre>
'''Sample GDB disassembling session:'''
<pre>
(gdb) disassemble /m loop
Dump of assembler code for function loop:
9 void loop() {
0x00000000004004c7 <+0>: push %rbp
0x00000000004004c8 <+1>: mov %rsp,%rbp

10 int v[2];
11 int i, k;
12
13 for(i = 0; i < 8; i++) {
0x00000000004004cb <+4>: movl $0x0,-0x4(%rbp)
0x00000000004004d2 <+11>: jmp 0x4004f5 <loop+46>
0x00000000004004f1 <+42>: addl $0x1,-0x4(%rbp)
0x00000000004004f5 <+46>: cmpl $0x7,-0x4(%rbp)
0x00000000004004f9 <+50>: jle 0x4004d4 <loop+13>

14 k = i/2*2; /* should have been k = i/(2*2); */
0x00000000004004d4 <+13>: mov -0x4(%rbp),%eax
0x00000000004004d7 <+16>: mov %eax,%edx
0x00000000004004d9 <+18>: shr $0x1f,%edx
0x00000000004004dc <+21>: add %edx,%eax
0x00000000004004de <+23>: sar %eax
0x00000000004004e0 <+25>: add %eax,%eax
0x00000000004004e2 <+27>: mov %eax,-0x8(%rbp)

15 v[k] = i;
0x00000000004004e5 <+30>: mov -0x8(%rbp),%eax
0x00000000004004e8 <+33>: cltq
0x00000000004004ea <+35>: mov -0x4(%rbp),%edx
0x00000000004004ed <+38>: mov %edx,-0x10(%rbp,%rax,4)

16 }
17 }
0x00000000004004fb <+52>: nop
0x00000000004004fc <+53>: pop %rbp
0x00000000004004fd <+54>: retq

End of assembler dump.
</pre>
'''Sample objdump disassembling session:'''
<pre>
$ objdump -S -D bug
[...]
00000000004004c7 <loop>:

void loop() {
4004c7: 55 push %rbp
4004c8: 48 89 e5 mov %rsp,%rbp
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
4004cb: c7 45 fc 00 00 00 00 movl $0x0,-0x4(%rbp)
4004d2: eb 21 jmp 4004f5 <loop+0x2e>
k = i/2*2; /* should have been k = i/(2*2); */
4004d4: 8b 45 fc mov -0x4(%rbp),%eax
4004d7: 89 c2 mov %eax,%edx
4004d9: c1 ea 1f shr $0x1f,%edx
4004dc: 01 d0 add %edx,%eax
4004de: d1 f8 sar %eax
4004e0: 01 c0 add %eax,%eax
4004e2: 89 45 f8 mov %eax,-0x8(%rbp)
v[k] = i;
4004e5: 8b 45 f8 mov -0x8(%rbp),%eax
4004e8: 48 98 cltq
4004ea: 8b 55 fc mov -0x4(%rbp),%edx
4004ed: 89 54 85 f0 mov %edx,-0x10(%rbp,%rax,4)

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
4004f1: 83 45 fc 01 addl $0x1,-0x4(%rbp)
4004f5: 83 7d fc 07 cmpl $0x7,-0x4(%rbp)
4004f9: 7e d9 jle 4004d4 <loop+0xd>
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}
4004fb: 90 nop
4004fc: 5d pop %rbp
4004fd: c3 retq
[...]
</pre>

[[Category:debugger software]][[Category:bwUniCluster]]

Development/GDB

2016-02-24T08:42:03Z

R Doros: /* Disassembling */ More Examples

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| devel/gdb
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]]
|-
| License
| [http://www.gnu.org/licenses/ GPL]
|-
|Citing
| n/a
|-
| Links
| [http://www.gnu.org/software/gdb/ Homepage] | [http://www.gnu.org/software/gdb/documentation/ Documentation] | [https://sourceware.org/gdb/wiki/ Wiki] | [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists]
|-
| Graphical Interface
| No
|-
| Included modules
| icc | icpc | ifort | idb
|}
 
= Introduction =
The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/devel/gdb
|width=99%
|height=120
}}
On the command line interface of any bwHPC cluster you'll get a list of available versions
by using the command ''''module avail devel/gdb''''.
<pre>
$ : bwUniCluster
$ module avail devel/gdb
------------------------ /opt/bwhpc/common/modulefiles -------------------------
devel/gdb/7.7</pre>

Attention! 
The default Linux GDB may be a different version.
Check the default version by: ''''gdb --version'''' without loaded GDB-module.
<pre>
$ module clear # clear all loaded modules
Are you sure you want to clear all loaded modules!? [n] y
$ gdb --version # this is the default version
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-83.el6)
[...]
$ module avail devel/gdb # available GDB-module
------------------------ /opt/bwhpc/common/modulefiles -------------------------
devel/gdb/7.7
</pre>
''It is not recommended (or impossible) the debug MPI-based binaries with GDB. You can't attach to MPI'ed jobs, too.''
 
 

= Loading =
It is possible to choose between several versions of GDB. By default both programs are installed. You can check the version which is currently used with:
<pre>$ gdb --version</pre>
<pre>$ gcc --version</pre>
 
To use a different version of GCC or GDB, you have to load it through the module system. For example you need GDB 7.7 or higher for GCC version 4.8 and above. In this case load the devel/gdb module.
 
How to load the desired version (e.g. GDB version 7.7):
<pre>$ module load devel/gdb/7.7</pre>
 
If you want to use IDB load the Intel compiler module:
<pre>$ module load compiler/intel</pre>
 
= Documentation =
For '''online documentation''' see the links section in the summary table at the top of this page. For '''local documentation''' consult the man page.
<pre>$ man gdb</pre>
or
<pre>$ man idb</pre>
 
= Basic commands =
The code you want to debug should be compiled with the -g option. If the optimization flag is not set, the GCC will still do some basic optimization. Therefore, it is recommended to turn off the optmization explicitly with the -O0 parameter for debugging. To start a debug session for a program execute GDB with the program path as parameter:
<pre>$ gdb ./example</pre>

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

{| width=600px class="wikitable"
|-
! Command
! Description
|-
| help ''cmd''
| Show help for command ''cmd''.
|-
| break ''func''
| Set a breakpoint at function ''func''.
|-
| run
| Start program.
|-
| next
| Go to next program line. Do not enter functions.
|-
| step
| Go to next program line. Enter functions.
|-
| list
| Show the surrounding source code of the currently processed line.
|-
| print ''expr''
| Print the value of the expression ''expr''.
|-
| display ''expr''
| Display the value of the expression ''expr'' every time the program stops.
|-
| watch ''expr''
| Stop when value of the expression ''expr'' changes.
|-
| continue
| Continue execution until a breakpoint or a watchpoint appears.
|-
| backtrace
| Print a list of functions that are currently active.
|-
| quit
| Exit GDB.
|}

'''Example:'''
We debug the following program called bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>

int global = 0;

void begin() {
global = 1;
}

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}

void end() {
global = 2;
}

int main() {
begin();
loop();
end();

return 0;
}
</source>

'''Sample GDB session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27 loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
(gdb) next
27 loop();
(gdb) step
loop () at bug.c:13
13 for(i = 0; i < 8; i++)
(gdb) next
15 k = i/2*2;
(gdb) next
16 v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16 v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit
</pre>
 

= Core dumps =
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
<pre>$ ulimit -c unlimited</pre>
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
<pre>$ gdb ./ex core.xxx</pre>
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.
 
 

= Multithreaded debugging =
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step.
The following commands are useful for multithreaded debugging:

{| width=600px class="wikitable"
|-
! Command
! Description
|-
| info threads
| Shows the status of all existing threads.
|-
| thread ''num''
| Switches to the thread with the number ''num''
|}

'''Example:'''
We debug the following program called thread_bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>
#include <pthread.h>

pthread_t thread;

void* thread3 (void* d)
{
int w[2];
int c, l;

for(c = 0; c < 8; c++) {
l = c/2*2; /* should have been l = c/(2*2); */
w[l] = c;
}

return NULL;
}

void* thread2 (void* d)
{
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
sleep(4);
k = i/(2*2); /* should have been k = i/(2*2); */
v[k] = i;
}

return NULL;
}

int main (){

pthread_create (&thread, NULL, thread2, NULL);
pthread_create (&thread, NULL, thread3, NULL);

//Thread 1
int count1 = 0;

while(count1 < 4000){
printf("Thread 1: %d\n", count1++);
}

pthread_join(thread, NULL);
return 0;
}
</source>

'''Sample GDB thread session:'''
<pre>
$ gcc -g thread_bug.c -o thread_bug -lpthread
$ gdb ./thread_bug
[...]
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break thread3
Breakpoint 1 at 0x40060c: file thread_bug.c, line 11.
(gdb) break thread2
Breakpoint 2 at 0x400650: file thread_bug.c, line 24.
(gdb) break main
Breakpoint 3 at 0x40069e: file thread_bug.c, line 35.
(gdb) run
Starting program: /tank/home/doros/.t/thread_bug
[Thread debugging using libthread_db enabled]

Breakpoint 3, main () at thread_bug.c:35
35 pthread_create (&thread, NULL, thread2, NULL);
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64
(gdb) info threads
* 1 Thread 0x7ffff7fe5700 (LWP 28260) main () at thread_bug.c:35
(gdb) next
[New Thread 0x7ffff7fe3700 (LWP 28303)]
36 pthread_create (&thread, NULL, thread3, NULL);
(gdb) info threads
2 Thread 0x7ffff7fe3700 (LWP 28303) thread2 (d=0x0) at thread_bug.c:24
* 1 Thread 0x7ffff7fe5700 (LWP 28260) main () at thread_bug.c:36
(gdb) next
[Switching to Thread 0x7ffff7fe3700 (LWP 28303)]

Breakpoint 2, thread2 (d=0x0) at thread_bug.c:24
24 for(i = 0; i < 8; i++) {
(gdb) next
25 sleep(4);
(gdb) next
[New Thread 0x7ffff77e2700 (LWP 28344)]
[Switching to Thread 0x7ffff77e2700 (LWP 28344)]

Breakpoint 1, thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) info threads
* 3 Thread 0x7ffff77e2700 (LWP 28344) thread3 (d=0x0) at thread_bug.c:11
2 Thread 0x7ffff7fe3700 (LWP 28303) 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 28260) 0x000000362f8725db in _IO_new_file_overflow () from /lib64/libc.so.6
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff7fe3700 (LWP 28303))]#0 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
(gdb) next
Single stepping until exit from function nanosleep,
which has no line number information.
[Switching to Thread 0x7ffff77e2700 (LWP 28344)]

Breakpoint 1, thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff7fe3700 (LWP 28303))]#0 0x000000362f8acce9 in nanosleep () from /lib64/libc.so.6
(gdb) next
Single stepping until exit from function nanosleep,
which has no line number information.
0x000000362f8acb50 in sleep () from /lib64/libc.so.6
(gdb) info threads
3 Thread 0x7ffff77e2700 (LWP 28344) thread3 (d=0x0) at thread_bug.c:11
* 2 Thread 0x7ffff7fe3700 (LWP 28303) 0x000000362f8acb50 in sleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 28260) 0x000000362f8476f0 in vfprintf () from /lib64/libc.so.6
(gdb) thread 3
[Switching to thread 3 (Thread 0x7ffff77e2700 (LWP 28344))]#0 thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) next
12 l = c/2*2; /* should have been l = c/(2*2); */
(gdb) watch (k >= 2)
No symbol "k" in current context.
(gdb) watch (l >= 2)
Hardware watchpoint 4: (l >= 2)
(gdb) continue
Continuing.
Thread 1: 0
Thread 1: 1
Thread 1: 2
Thread 1: 3
Thread 1: 4
[...]
Hardware watchpoint 4: (l >= 2)

Old value = 0
New value = 1
thread3 (d=0x0) at thread_bug.c:13
13 w[l] = c;
(gdb) print l
$1 = 2
(gdb) print c
$2 = 2
(gdb) quit
</pre>
= Disassembling =
{| width=600px class="wikitable"
|-
! Command
! Description
|-
| info functions
| Shows names and data types of all defined functions.
|-
| info line "function"
| Map source lines to memory adresses (and back).
|-
| disassemble ''function''
| Disassembles "function" (or a function fragment).
|}
'''Sample GDB disassembling session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
[...]
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bug...done.
(gdb) info functions
All defined functions:

File bug.c:
void begin();
void end();
void loop();
int main();

Non-debugging symbols:
0x0000000000400370 _init
0x00000000004003a0 __libc_start_main@plt
0x00000000004003b0 __gmon_start__@plt
0x00000000004003c0 _start
0x00000000004003f0 deregister_tm_clones
0x0000000000400430 register_tm_clones
0x0000000000400470 __do_global_dtors_aux
0x0000000000400490 frame_dummy
0x0000000000400540 __libc_csu_init
0x00000000004005b0 __libc_csu_fini
0x00000000004005b4 _fini
</pre>
'''Sample GDB disassembling session:'''
<pre>
(gdb) disassemble main
Dump of assembler code for function main:
0x000000000040050f <+0>: push %rbp
0x0000000000400510 <+1>: mov %rsp,%rbp
0x0000000000400513 <+4>: mov $0x0,%eax
0x0000000000400518 <+9>: callq 0x4004b6 <begin>
0x000000000040051d <+14>: mov $0x0,%eax
0x0000000000400522 <+19>: callq 0x4004c7 <loop>
0x0000000000400527 <+24>: mov $0x0,%eax
0x000000000040052c <+29>: callq 0x4004fe <end>
0x0000000000400531 <+34>: mov $0x0,%eax
0x0000000000400536 <+39>: pop %rbp
0x0000000000400537 <+40>: retq
End of assembler dump.
</pre>
'''Sample GDB disassembling session:'''
<pre>
(gdb) disassemble /m main
Dump of assembler code for function main:
23 int main() {
0x000000000040050f <+0>: push %rbp
0x0000000000400510 <+1>: mov %rsp,%rbp

24 begin();
0x0000000000400513 <+4>: mov $0x0,%eax
0x0000000000400518 <+9>: callq 0x4004b6 <begin>

25 loop();
0x000000000040051d <+14>: mov $0x0,%eax
0x0000000000400522 <+19>: callq 0x4004c7 <loop>

26 end();
0x0000000000400527 <+24>: mov $0x0,%eax
0x000000000040052c <+29>: callq 0x4004fe <end>

27
28 return 0;
0x0000000000400531 <+34>: mov $0x0,%eax

29 }
0x0000000000400536 <+39>: pop %rbp
0x0000000000400537 <+40>: retq

End of assembler dump.
</pre>

[[Category:debugger software]][[Category:bwUniCluster]]

Development/GDB

2016-02-24T08:36:38Z

R Doros: First disassembling session

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| devel/gdb
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]]
|-
| License
| [http://www.gnu.org/licenses/ GPL]
|-
|Citing
| n/a
|-
| Links
| [http://www.gnu.org/software/gdb/ Homepage] | [http://www.gnu.org/software/gdb/documentation/ Documentation] | [https://sourceware.org/gdb/wiki/ Wiki] | [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists]
|-
| Graphical Interface
| No
|-
| Included modules
| icc | icpc | ifort | idb
|}
 
= Introduction =
The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/devel/gdb
|width=99%
|height=120
}}
On the command line interface of any bwHPC cluster you'll get a list of available versions
by using the command ''''module avail devel/gdb''''.
<pre>
$ : bwUniCluster
$ module avail devel/gdb
------------------------ /opt/bwhpc/common/modulefiles -------------------------
devel/gdb/7.7</pre>

Attention! 
The default Linux GDB may be a different version.
Check the default version by: ''''gdb --version'''' without loaded GDB-module.
<pre>
$ module clear # clear all loaded modules
Are you sure you want to clear all loaded modules!? [n] y
$ gdb --version # this is the default version
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-83.el6)
[...]
$ module avail devel/gdb # available GDB-module
------------------------ /opt/bwhpc/common/modulefiles -------------------------
devel/gdb/7.7
</pre>
''It is not recommended (or impossible) the debug MPI-based binaries with GDB. You can't attach to MPI'ed jobs, too.''
 
 

= Loading =
It is possible to choose between several versions of GDB. By default both programs are installed. You can check the version which is currently used with:
<pre>$ gdb --version</pre>
<pre>$ gcc --version</pre>
 
To use a different version of GCC or GDB, you have to load it through the module system. For example you need GDB 7.7 or higher for GCC version 4.8 and above. In this case load the devel/gdb module.
 
How to load the desired version (e.g. GDB version 7.7):
<pre>$ module load devel/gdb/7.7</pre>
 
If you want to use IDB load the Intel compiler module:
<pre>$ module load compiler/intel</pre>
 
= Documentation =
For '''online documentation''' see the links section in the summary table at the top of this page. For '''local documentation''' consult the man page.
<pre>$ man gdb</pre>
or
<pre>$ man idb</pre>
 
= Basic commands =
The code you want to debug should be compiled with the -g option. If the optimization flag is not set, the GCC will still do some basic optimization. Therefore, it is recommended to turn off the optmization explicitly with the -O0 parameter for debugging. To start a debug session for a program execute GDB with the program path as parameter:
<pre>$ gdb ./example</pre>

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

{| width=600px class="wikitable"
|-
! Command
! Description
|-
| help ''cmd''
| Show help for command ''cmd''.
|-
| break ''func''
| Set a breakpoint at function ''func''.
|-
| run
| Start program.
|-
| next
| Go to next program line. Do not enter functions.
|-
| step
| Go to next program line. Enter functions.
|-
| list
| Show the surrounding source code of the currently processed line.
|-
| print ''expr''
| Print the value of the expression ''expr''.
|-
| display ''expr''
| Display the value of the expression ''expr'' every time the program stops.
|-
| watch ''expr''
| Stop when value of the expression ''expr'' changes.
|-
| continue
| Continue execution until a breakpoint or a watchpoint appears.
|-
| backtrace
| Print a list of functions that are currently active.
|-
| quit
| Exit GDB.
|}

'''Example:'''
We debug the following program called bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>

int global = 0;

void begin() {
global = 1;
}

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}

void end() {
global = 2;
}

int main() {
begin();
loop();
end();

return 0;
}
</source>

'''Sample GDB session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27 loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
(gdb) next
27 loop();
(gdb) step
loop () at bug.c:13
13 for(i = 0; i < 8; i++)
(gdb) next
15 k = i/2*2;
(gdb) next
16 v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16 v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit
</pre>
 

= Core dumps =
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
<pre>$ ulimit -c unlimited</pre>
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
<pre>$ gdb ./ex core.xxx</pre>
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.
 
 

= Multithreaded debugging =
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step.
The following commands are useful for multithreaded debugging:

{| width=600px class="wikitable"
|-
! Command
! Description
|-
| info threads
| Shows the status of all existing threads.
|-
| thread ''num''
| Switches to the thread with the number ''num''
|}

'''Example:'''
We debug the following program called thread_bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>
#include <pthread.h>

pthread_t thread;

void* thread3 (void* d)
{
int w[2];
int c, l;

for(c = 0; c < 8; c++) {
l = c/2*2; /* should have been l = c/(2*2); */
w[l] = c;
}

return NULL;
}

void* thread2 (void* d)
{
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
sleep(4);
k = i/(2*2); /* should have been k = i/(2*2); */
v[k] = i;
}

return NULL;
}

int main (){

pthread_create (&thread, NULL, thread2, NULL);
pthread_create (&thread, NULL, thread3, NULL);

//Thread 1
int count1 = 0;

while(count1 < 4000){
printf("Thread 1: %d\n", count1++);
}

pthread_join(thread, NULL);
return 0;
}
</source>

'''Sample GDB thread session:'''
<pre>
$ gcc -g thread_bug.c -o thread_bug -lpthread
$ gdb ./thread_bug
[...]
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break thread3
Breakpoint 1 at 0x40060c: file thread_bug.c, line 11.
(gdb) break thread2
Breakpoint 2 at 0x400650: file thread_bug.c, line 24.
(gdb) break main
Breakpoint 3 at 0x40069e: file thread_bug.c, line 35.
(gdb) run
Starting program: /tank/home/doros/.t/thread_bug
[Thread debugging using libthread_db enabled]

Breakpoint 3, main () at thread_bug.c:35
35 pthread_create (&thread, NULL, thread2, NULL);
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64
(gdb) info threads
* 1 Thread 0x7ffff7fe5700 (LWP 28260) main () at thread_bug.c:35
(gdb) next
[New Thread 0x7ffff7fe3700 (LWP 28303)]
36 pthread_create (&thread, NULL, thread3, NULL);
(gdb) info threads
2 Thread 0x7ffff7fe3700 (LWP 28303) thread2 (d=0x0) at thread_bug.c:24
* 1 Thread 0x7ffff7fe5700 (LWP 28260) main () at thread_bug.c:36
(gdb) next
[Switching to Thread 0x7ffff7fe3700 (LWP 28303)]

Breakpoint 2, thread2 (d=0x0) at thread_bug.c:24
24 for(i = 0; i < 8; i++) {
(gdb) next
25 sleep(4);
(gdb) next
[New Thread 0x7ffff77e2700 (LWP 28344)]
[Switching to Thread 0x7ffff77e2700 (LWP 28344)]

Breakpoint 1, thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) info threads
* 3 Thread 0x7ffff77e2700 (LWP 28344) thread3 (d=0x0) at thread_bug.c:11
2 Thread 0x7ffff7fe3700 (LWP 28303) 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 28260) 0x000000362f8725db in _IO_new_file_overflow () from /lib64/libc.so.6
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff7fe3700 (LWP 28303))]#0 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
(gdb) next
Single stepping until exit from function nanosleep,
which has no line number information.
[Switching to Thread 0x7ffff77e2700 (LWP 28344)]

Breakpoint 1, thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff7fe3700 (LWP 28303))]#0 0x000000362f8acce9 in nanosleep () from /lib64/libc.so.6
(gdb) next
Single stepping until exit from function nanosleep,
which has no line number information.
0x000000362f8acb50 in sleep () from /lib64/libc.so.6
(gdb) info threads
3 Thread 0x7ffff77e2700 (LWP 28344) thread3 (d=0x0) at thread_bug.c:11
* 2 Thread 0x7ffff7fe3700 (LWP 28303) 0x000000362f8acb50 in sleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 28260) 0x000000362f8476f0 in vfprintf () from /lib64/libc.so.6
(gdb) thread 3
[Switching to thread 3 (Thread 0x7ffff77e2700 (LWP 28344))]#0 thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) next
12 l = c/2*2; /* should have been l = c/(2*2); */
(gdb) watch (k >= 2)
No symbol "k" in current context.
(gdb) watch (l >= 2)
Hardware watchpoint 4: (l >= 2)
(gdb) continue
Continuing.
Thread 1: 0
Thread 1: 1
Thread 1: 2
Thread 1: 3
Thread 1: 4
[...]
Hardware watchpoint 4: (l >= 2)

Old value = 0
New value = 1
thread3 (d=0x0) at thread_bug.c:13
13 w[l] = c;
(gdb) print l
$1 = 2
(gdb) print c
$2 = 2
(gdb) quit
</pre>
= Disassembling =
{| width=600px class="wikitable"
|-
! Command
! Description
|-
| info functions
| Shows names and data types of all defined functions.
|-
| info line "function"
| Map source lines to memory adresses (and back).
|-
| disassemble ''function''
| Disassembles "function" (or a function fragment).
|}
'''Sample GDB disassembling session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
[...]
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bug...done.
(gdb) info functions
All defined functions:

File bug.c:
void begin();
void end();
void loop();
int main();

Non-debugging symbols:
0x0000000000400370 _init
0x00000000004003a0 __libc_start_main@plt
0x00000000004003b0 __gmon_start__@plt
0x00000000004003c0 _start
0x00000000004003f0 deregister_tm_clones
0x0000000000400430 register_tm_clones
0x0000000000400470 __do_global_dtors_aux
0x0000000000400490 frame_dummy
0x0000000000400540 __libc_csu_init
0x00000000004005b0 __libc_csu_fini
0x00000000004005b4 _fini
</pre>
[[Category:debugger software]][[Category:bwUniCluster]]

Development/GDB

2016-02-24T08:30:48Z

R Doros: /* Disassembling */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| devel/gdb
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]]
|-
| License
| [http://www.gnu.org/licenses/ GPL]
|-
|Citing
| n/a
|-
| Links
| [http://www.gnu.org/software/gdb/ Homepage] | [http://www.gnu.org/software/gdb/documentation/ Documentation] | [https://sourceware.org/gdb/wiki/ Wiki] | [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists]
|-
| Graphical Interface
| No
|-
| Included modules
| icc | icpc | ifort | idb
|}
 
= Introduction =
The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/devel/gdb
|width=99%
|height=120
}}
On the command line interface of any bwHPC cluster you'll get a list of available versions
by using the command ''''module avail devel/gdb''''.
<pre>
$ : bwUniCluster
$ module avail devel/gdb
------------------------ /opt/bwhpc/common/modulefiles -------------------------
devel/gdb/7.7</pre>

Attention! 
The default Linux GDB may be a different version.
Check the default version by: ''''gdb --version'''' without loaded GDB-module.
<pre>
$ module clear # clear all loaded modules
Are you sure you want to clear all loaded modules!? [n] y
$ gdb --version # this is the default version
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-83.el6)
[...]
$ module avail devel/gdb # available GDB-module
------------------------ /opt/bwhpc/common/modulefiles -------------------------
devel/gdb/7.7
</pre>
''It is not recommended (or impossible) the debug MPI-based binaries with GDB. You can't attach to MPI'ed jobs, too.''
 
 

= Loading =
It is possible to choose between several versions of GDB. By default both programs are installed. You can check the version which is currently used with:
<pre>$ gdb --version</pre>
<pre>$ gcc --version</pre>
 
To use a different version of GCC or GDB, you have to load it through the module system. For example you need GDB 7.7 or higher for GCC version 4.8 and above. In this case load the devel/gdb module.
 
How to load the desired version (e.g. GDB version 7.7):
<pre>$ module load devel/gdb/7.7</pre>
 
If you want to use IDB load the Intel compiler module:
<pre>$ module load compiler/intel</pre>
 
= Documentation =
For '''online documentation''' see the links section in the summary table at the top of this page. For '''local documentation''' consult the man page.
<pre>$ man gdb</pre>
or
<pre>$ man idb</pre>
 
= Basic commands =
The code you want to debug should be compiled with the -g option. If the optimization flag is not set, the GCC will still do some basic optimization. Therefore, it is recommended to turn off the optmization explicitly with the -O0 parameter for debugging. To start a debug session for a program execute GDB with the program path as parameter:
<pre>$ gdb ./example</pre>

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

{| width=600px class="wikitable"
|-
! Command
! Description
|-
| help ''cmd''
| Show help for command ''cmd''.
|-
| break ''func''
| Set a breakpoint at function ''func''.
|-
| run
| Start program.
|-
| next
| Go to next program line. Do not enter functions.
|-
| step
| Go to next program line. Enter functions.
|-
| list
| Show the surrounding source code of the currently processed line.
|-
| print ''expr''
| Print the value of the expression ''expr''.
|-
| display ''expr''
| Display the value of the expression ''expr'' every time the program stops.
|-
| watch ''expr''
| Stop when value of the expression ''expr'' changes.
|-
| continue
| Continue execution until a breakpoint or a watchpoint appears.
|-
| backtrace
| Print a list of functions that are currently active.
|-
| quit
| Exit GDB.
|}

'''Example:'''
We debug the following program called bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>

int global = 0;

void begin() {
global = 1;
}

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}

void end() {
global = 2;
}

int main() {
begin();
loop();
end();

return 0;
}
</source>

'''Sample GDB session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27 loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
(gdb) next
27 loop();
(gdb) step
loop () at bug.c:13
13 for(i = 0; i < 8; i++)
(gdb) next
15 k = i/2*2;
(gdb) next
16 v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16 v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit
</pre>
 

= Core dumps =
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
<pre>$ ulimit -c unlimited</pre>
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
<pre>$ gdb ./ex core.xxx</pre>
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.
 
 

= Multithreaded debugging =
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step.
The following commands are useful for multithreaded debugging:

{| width=600px class="wikitable"
|-
! Command
! Description
|-
| info threads
| Shows the status of all existing threads.
|-
| thread ''num''
| Switches to the thread with the number ''num''
|}

'''Example:'''
We debug the following program called thread_bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>
#include <pthread.h>

pthread_t thread;

void* thread3 (void* d)
{
int w[2];
int c, l;

for(c = 0; c < 8; c++) {
l = c/2*2; /* should have been l = c/(2*2); */
w[l] = c;
}

return NULL;
}

void* thread2 (void* d)
{
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
sleep(4);
k = i/(2*2); /* should have been k = i/(2*2); */
v[k] = i;
}

return NULL;
}

int main (){

pthread_create (&thread, NULL, thread2, NULL);
pthread_create (&thread, NULL, thread3, NULL);

//Thread 1
int count1 = 0;

while(count1 < 4000){
printf("Thread 1: %d\n", count1++);
}

pthread_join(thread, NULL);
return 0;
}
</source>

'''Sample GDB thread session:'''
<pre>
$ gcc -g thread_bug.c -o thread_bug -lpthread
$ gdb ./thread_bug
[...]
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break thread3
Breakpoint 1 at 0x40060c: file thread_bug.c, line 11.
(gdb) break thread2
Breakpoint 2 at 0x400650: file thread_bug.c, line 24.
(gdb) break main
Breakpoint 3 at 0x40069e: file thread_bug.c, line 35.
(gdb) run
Starting program: /tank/home/doros/.t/thread_bug
[Thread debugging using libthread_db enabled]

Breakpoint 3, main () at thread_bug.c:35
35 pthread_create (&thread, NULL, thread2, NULL);
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64
(gdb) info threads
* 1 Thread 0x7ffff7fe5700 (LWP 28260) main () at thread_bug.c:35
(gdb) next
[New Thread 0x7ffff7fe3700 (LWP 28303)]
36 pthread_create (&thread, NULL, thread3, NULL);
(gdb) info threads
2 Thread 0x7ffff7fe3700 (LWP 28303) thread2 (d=0x0) at thread_bug.c:24
* 1 Thread 0x7ffff7fe5700 (LWP 28260) main () at thread_bug.c:36
(gdb) next
[Switching to Thread 0x7ffff7fe3700 (LWP 28303)]

Breakpoint 2, thread2 (d=0x0) at thread_bug.c:24
24 for(i = 0; i < 8; i++) {
(gdb) next
25 sleep(4);
(gdb) next
[New Thread 0x7ffff77e2700 (LWP 28344)]
[Switching to Thread 0x7ffff77e2700 (LWP 28344)]

Breakpoint 1, thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) info threads
* 3 Thread 0x7ffff77e2700 (LWP 28344) thread3 (d=0x0) at thread_bug.c:11
2 Thread 0x7ffff7fe3700 (LWP 28303) 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 28260) 0x000000362f8725db in _IO_new_file_overflow () from /lib64/libc.so.6
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff7fe3700 (LWP 28303))]#0 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
(gdb) next
Single stepping until exit from function nanosleep,
which has no line number information.
[Switching to Thread 0x7ffff77e2700 (LWP 28344)]

Breakpoint 1, thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff7fe3700 (LWP 28303))]#0 0x000000362f8acce9 in nanosleep () from /lib64/libc.so.6
(gdb) next
Single stepping until exit from function nanosleep,
which has no line number information.
0x000000362f8acb50 in sleep () from /lib64/libc.so.6
(gdb) info threads
3 Thread 0x7ffff77e2700 (LWP 28344) thread3 (d=0x0) at thread_bug.c:11
* 2 Thread 0x7ffff7fe3700 (LWP 28303) 0x000000362f8acb50 in sleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 28260) 0x000000362f8476f0 in vfprintf () from /lib64/libc.so.6
(gdb) thread 3
[Switching to thread 3 (Thread 0x7ffff77e2700 (LWP 28344))]#0 thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) next
12 l = c/2*2; /* should have been l = c/(2*2); */
(gdb) watch (k >= 2)
No symbol "k" in current context.
(gdb) watch (l >= 2)
Hardware watchpoint 4: (l >= 2)
(gdb) continue
Continuing.
Thread 1: 0
Thread 1: 1
Thread 1: 2
Thread 1: 3
Thread 1: 4
[...]
Hardware watchpoint 4: (l >= 2)

Old value = 0
New value = 1
thread3 (d=0x0) at thread_bug.c:13
13 w[l] = c;
(gdb) print l
$1 = 2
(gdb) print c
$2 = 2
(gdb) quit
</pre>
= Disassembling =
{| width=600px class="wikitable"
|-
! Command
! Description
|-
| info functions
| Shows names and data types of all defined functions.
|-
| info line "function"
| Map source lines to memory adresses (and back).
|-
| disassemble ''function''
| Disassembles "function" (or a function fragment).
|}
'''Sample GDB disassembling session:'''
<pre>

</pre>
[[Category:debugger software]][[Category:bwUniCluster]]

Development/GDB

2016-02-23T15:40:08Z

R Doros: /* Disassembling */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| devel/gdb
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]]
|-
| License
| [http://www.gnu.org/licenses/ GPL]
|-
|Citing
| n/a
|-
| Links
| [http://www.gnu.org/software/gdb/ Homepage] | [http://www.gnu.org/software/gdb/documentation/ Documentation] | [https://sourceware.org/gdb/wiki/ Wiki] | [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists]
|-
| Graphical Interface
| No
|-
| Included modules
| icc | icpc | ifort | idb
|}
 
= Introduction =
The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/devel/gdb
|width=99%
|height=120
}}
On the command line interface of any bwHPC cluster you'll get a list of available versions
by using the command ''''module avail devel/gdb''''.
<pre>
$ : bwUniCluster
$ module avail devel/gdb
------------------------ /opt/bwhpc/common/modulefiles -------------------------
devel/gdb/7.7</pre>

Attention! 
The default Linux GDB may be a different version.
Check the default version by: ''''gdb --version'''' without loaded GDB-module.
<pre>
$ module clear # clear all loaded modules
Are you sure you want to clear all loaded modules!? [n] y
$ gdb --version # this is the default version
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-83.el6)
[...]
$ module avail devel/gdb # available GDB-module
------------------------ /opt/bwhpc/common/modulefiles -------------------------
devel/gdb/7.7
</pre>
''It is not recommended (or impossible) the debug MPI-based binaries with GDB. You can't attach to MPI'ed jobs, too.''
 
 

= Loading =
It is possible to choose between several versions of GDB. By default both programs are installed. You can check the version which is currently used with:
<pre>$ gdb --version</pre>
<pre>$ gcc --version</pre>
 
To use a different version of GCC or GDB, you have to load it through the module system. For example you need GDB 7.7 or higher for GCC version 4.8 and above. In this case load the devel/gdb module.
 
How to load the desired version (e.g. GDB version 7.7):
<pre>$ module load devel/gdb/7.7</pre>
 
If you want to use IDB load the Intel compiler module:
<pre>$ module load compiler/intel</pre>
 
= Documentation =
For '''online documentation''' see the links section in the summary table at the top of this page. For '''local documentation''' consult the man page.
<pre>$ man gdb</pre>
or
<pre>$ man idb</pre>
 
= Basic commands =
The code you want to debug should be compiled with the -g option. If the optimization flag is not set, the GCC will still do some basic optimization. Therefore, it is recommended to turn off the optmization explicitly with the -O0 parameter for debugging. To start a debug session for a program execute GDB with the program path as parameter:
<pre>$ gdb ./example</pre>

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

{| width=600px class="wikitable"
|-
! Command
! Description
|-
| help ''cmd''
| Show help for command ''cmd''.
|-
| break ''func''
| Set a breakpoint at function ''func''.
|-
| run
| Start program.
|-
| next
| Go to next program line. Do not enter functions.
|-
| step
| Go to next program line. Enter functions.
|-
| list
| Show the surrounding source code of the currently processed line.
|-
| print ''expr''
| Print the value of the expression ''expr''.
|-
| display ''expr''
| Display the value of the expression ''expr'' every time the program stops.
|-
| watch ''expr''
| Stop when value of the expression ''expr'' changes.
|-
| continue
| Continue execution until a breakpoint or a watchpoint appears.
|-
| backtrace
| Print a list of functions that are currently active.
|-
| quit
| Exit GDB.
|}

'''Example:'''
We debug the following program called bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>

int global = 0;

void begin() {
global = 1;
}

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}

void end() {
global = 2;
}

int main() {
begin();
loop();
end();

return 0;
}
</source>

'''Sample GDB session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27 loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
(gdb) next
27 loop();
(gdb) step
loop () at bug.c:13
13 for(i = 0; i < 8; i++)
(gdb) next
15 k = i/2*2;
(gdb) next
16 v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16 v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit
</pre>
 

= Core dumps =
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
<pre>$ ulimit -c unlimited</pre>
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
<pre>$ gdb ./ex core.xxx</pre>
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.
 
 

= Multithreaded debugging =
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step.
The following commands are useful for multithreaded debugging:

{| width=600px class="wikitable"
|-
! Command
! Description
|-
| info threads
| Shows the status of all existing threads.
|-
| thread ''num''
| Switches to the thread with the number ''num''
|}

'''Example:'''
We debug the following program called thread_bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>
#include <pthread.h>

pthread_t thread;

void* thread3 (void* d)
{
int w[2];
int c, l;

for(c = 0; c < 8; c++) {
l = c/2*2; /* should have been l = c/(2*2); */
w[l] = c;
}

return NULL;
}

void* thread2 (void* d)
{
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
sleep(4);
k = i/(2*2); /* should have been k = i/(2*2); */
v[k] = i;
}

return NULL;
}

int main (){

pthread_create (&thread, NULL, thread2, NULL);
pthread_create (&thread, NULL, thread3, NULL);

//Thread 1
int count1 = 0;

while(count1 < 4000){
printf("Thread 1: %d\n", count1++);
}

pthread_join(thread, NULL);
return 0;
}
</source>

'''Sample GDB thread session:'''
<pre>
$ gcc -g thread_bug.c -o thread_bug -lpthread
$ gdb ./thread_bug
[...]
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break thread3
Breakpoint 1 at 0x40060c: file thread_bug.c, line 11.
(gdb) break thread2
Breakpoint 2 at 0x400650: file thread_bug.c, line 24.
(gdb) break main
Breakpoint 3 at 0x40069e: file thread_bug.c, line 35.
(gdb) run
Starting program: /tank/home/doros/.t/thread_bug
[Thread debugging using libthread_db enabled]

Breakpoint 3, main () at thread_bug.c:35
35 pthread_create (&thread, NULL, thread2, NULL);
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64
(gdb) info threads
* 1 Thread 0x7ffff7fe5700 (LWP 28260) main () at thread_bug.c:35
(gdb) next
[New Thread 0x7ffff7fe3700 (LWP 28303)]
36 pthread_create (&thread, NULL, thread3, NULL);
(gdb) info threads
2 Thread 0x7ffff7fe3700 (LWP 28303) thread2 (d=0x0) at thread_bug.c:24
* 1 Thread 0x7ffff7fe5700 (LWP 28260) main () at thread_bug.c:36
(gdb) next
[Switching to Thread 0x7ffff7fe3700 (LWP 28303)]

Breakpoint 2, thread2 (d=0x0) at thread_bug.c:24
24 for(i = 0; i < 8; i++) {
(gdb) next
25 sleep(4);
(gdb) next
[New Thread 0x7ffff77e2700 (LWP 28344)]
[Switching to Thread 0x7ffff77e2700 (LWP 28344)]

Breakpoint 1, thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) info threads
* 3 Thread 0x7ffff77e2700 (LWP 28344) thread3 (d=0x0) at thread_bug.c:11
2 Thread 0x7ffff7fe3700 (LWP 28303) 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 28260) 0x000000362f8725db in _IO_new_file_overflow () from /lib64/libc.so.6
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff7fe3700 (LWP 28303))]#0 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
(gdb) next
Single stepping until exit from function nanosleep,
which has no line number information.
[Switching to Thread 0x7ffff77e2700 (LWP 28344)]

Breakpoint 1, thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff7fe3700 (LWP 28303))]#0 0x000000362f8acce9 in nanosleep () from /lib64/libc.so.6
(gdb) next
Single stepping until exit from function nanosleep,
which has no line number information.
0x000000362f8acb50 in sleep () from /lib64/libc.so.6
(gdb) info threads
3 Thread 0x7ffff77e2700 (LWP 28344) thread3 (d=0x0) at thread_bug.c:11
* 2 Thread 0x7ffff7fe3700 (LWP 28303) 0x000000362f8acb50 in sleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 28260) 0x000000362f8476f0 in vfprintf () from /lib64/libc.so.6
(gdb) thread 3
[Switching to thread 3 (Thread 0x7ffff77e2700 (LWP 28344))]#0 thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) next
12 l = c/2*2; /* should have been l = c/(2*2); */
(gdb) watch (k >= 2)
No symbol "k" in current context.
(gdb) watch (l >= 2)
Hardware watchpoint 4: (l >= 2)
(gdb) continue
Continuing.
Thread 1: 0
Thread 1: 1
Thread 1: 2
Thread 1: 3
Thread 1: 4
[...]
Hardware watchpoint 4: (l >= 2)

Old value = 0
New value = 1
thread3 (d=0x0) at thread_bug.c:13
13 w[l] = c;
(gdb) print l
$1 = 2
(gdb) print c
$2 = 2
(gdb) quit
</pre>
= Disassembling =
{| width=600px class="wikitable"
|-
! Command
! Description
|-
| info functions
| Shows names and data types of all defined functions.
|-
| info line "function"
| Map source lines to memory adresses (and back).
|-
| disassemble ''function''
| Disassembles "function" (or a function fragment).
|}

[[Category:debugger software]][[Category:bwUniCluster]]

Development/GDB

2016-02-23T15:25:16Z

R Doros: Start new chapter

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| devel/gdb
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]]
|-
| License
| [http://www.gnu.org/licenses/ GPL]
|-
|Citing
| n/a
|-
| Links
| [http://www.gnu.org/software/gdb/ Homepage] | [http://www.gnu.org/software/gdb/documentation/ Documentation] | [https://sourceware.org/gdb/wiki/ Wiki] | [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists]
|-
| Graphical Interface
| No
|-
| Included modules
| icc | icpc | ifort | idb
|}
 
= Introduction =
The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/devel/gdb
|width=99%
|height=120
}}
On the command line interface of any bwHPC cluster you'll get a list of available versions
by using the command ''''module avail devel/gdb''''.
<pre>
$ : bwUniCluster
$ module avail devel/gdb
------------------------ /opt/bwhpc/common/modulefiles -------------------------
devel/gdb/7.7</pre>

Attention! 
The default Linux GDB may be a different version.
Check the default version by: ''''gdb --version'''' without loaded GDB-module.
<pre>
$ module clear # clear all loaded modules
Are you sure you want to clear all loaded modules!? [n] y
$ gdb --version # this is the default version
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-83.el6)
[...]
$ module avail devel/gdb # available GDB-module
------------------------ /opt/bwhpc/common/modulefiles -------------------------
devel/gdb/7.7
</pre>
''It is not recommended (or impossible) the debug MPI-based binaries with GDB. You can't attach to MPI'ed jobs, too.''
 
 

= Loading =
It is possible to choose between several versions of GDB. By default both programs are installed. You can check the version which is currently used with:
<pre>$ gdb --version</pre>
<pre>$ gcc --version</pre>
 
To use a different version of GCC or GDB, you have to load it through the module system. For example you need GDB 7.7 or higher for GCC version 4.8 and above. In this case load the devel/gdb module.
 
How to load the desired version (e.g. GDB version 7.7):
<pre>$ module load devel/gdb/7.7</pre>
 
If you want to use IDB load the Intel compiler module:
<pre>$ module load compiler/intel</pre>
 
= Documentation =
For '''online documentation''' see the links section in the summary table at the top of this page. For '''local documentation''' consult the man page.
<pre>$ man gdb</pre>
or
<pre>$ man idb</pre>
 
= Basic commands =
The code you want to debug should be compiled with the -g option. If the optimization flag is not set, the GCC will still do some basic optimization. Therefore, it is recommended to turn off the optmization explicitly with the -O0 parameter for debugging. To start a debug session for a program execute GDB with the program path as parameter:
<pre>$ gdb ./example</pre>

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

{| width=600px class="wikitable"
|-
! Command
! Description
|-
| help ''cmd''
| Show help for command ''cmd''.
|-
| break ''func''
| Set a breakpoint at function ''func''.
|-
| run
| Start program.
|-
| next
| Go to next program line. Do not enter functions.
|-
| step
| Go to next program line. Enter functions.
|-
| list
| Show the surrounding source code of the currently processed line.
|-
| print ''expr''
| Print the value of the expression ''expr''.
|-
| display ''expr''
| Display the value of the expression ''expr'' every time the program stops.
|-
| watch ''expr''
| Stop when value of the expression ''expr'' changes.
|-
| continue
| Continue execution until a breakpoint or a watchpoint appears.
|-
| backtrace
| Print a list of functions that are currently active.
|-
| quit
| Exit GDB.
|}

'''Example:'''
We debug the following program called bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>

int global = 0;

void begin() {
global = 1;
}

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}

void end() {
global = 2;
}

int main() {
begin();
loop();
end();

return 0;
}
</source>

'''Sample GDB session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27 loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
(gdb) next
27 loop();
(gdb) step
loop () at bug.c:13
13 for(i = 0; i < 8; i++)
(gdb) next
15 k = i/2*2;
(gdb) next
16 v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16 v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit
</pre>
 

= Core dumps =
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
<pre>$ ulimit -c unlimited</pre>
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
<pre>$ gdb ./ex core.xxx</pre>
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.
 
 

= Multithreaded debugging =
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step.
The following commands are useful for multithreaded debugging:

{| width=600px class="wikitable"
|-
! Command
! Description
|-
| info threads
| Shows the status of all existing threads.
|-
| thread ''num''
| Switches to the thread with the number ''num''
|}

'''Example:'''
We debug the following program called thread_bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>
#include <pthread.h>

pthread_t thread;

void* thread3 (void* d)
{
int w[2];
int c, l;

for(c = 0; c < 8; c++) {
l = c/2*2; /* should have been l = c/(2*2); */
w[l] = c;
}

return NULL;
}

void* thread2 (void* d)
{
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
sleep(4);
k = i/(2*2); /* should have been k = i/(2*2); */
v[k] = i;
}

return NULL;
}

int main (){

pthread_create (&thread, NULL, thread2, NULL);
pthread_create (&thread, NULL, thread3, NULL);

//Thread 1
int count1 = 0;

while(count1 < 4000){
printf("Thread 1: %d\n", count1++);
}

pthread_join(thread, NULL);
return 0;
}
</source>

'''Sample GDB thread session:'''
<pre>
$ gcc -g thread_bug.c -o thread_bug -lpthread
$ gdb ./thread_bug
[...]
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break thread3
Breakpoint 1 at 0x40060c: file thread_bug.c, line 11.
(gdb) break thread2
Breakpoint 2 at 0x400650: file thread_bug.c, line 24.
(gdb) break main
Breakpoint 3 at 0x40069e: file thread_bug.c, line 35.
(gdb) run
Starting program: /tank/home/doros/.t/thread_bug
[Thread debugging using libthread_db enabled]

Breakpoint 3, main () at thread_bug.c:35
35 pthread_create (&thread, NULL, thread2, NULL);
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64
(gdb) info threads
* 1 Thread 0x7ffff7fe5700 (LWP 28260) main () at thread_bug.c:35
(gdb) next
[New Thread 0x7ffff7fe3700 (LWP 28303)]
36 pthread_create (&thread, NULL, thread3, NULL);
(gdb) info threads
2 Thread 0x7ffff7fe3700 (LWP 28303) thread2 (d=0x0) at thread_bug.c:24
* 1 Thread 0x7ffff7fe5700 (LWP 28260) main () at thread_bug.c:36
(gdb) next
[Switching to Thread 0x7ffff7fe3700 (LWP 28303)]

Breakpoint 2, thread2 (d=0x0) at thread_bug.c:24
24 for(i = 0; i < 8; i++) {
(gdb) next
25 sleep(4);
(gdb) next
[New Thread 0x7ffff77e2700 (LWP 28344)]
[Switching to Thread 0x7ffff77e2700 (LWP 28344)]

Breakpoint 1, thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) info threads
* 3 Thread 0x7ffff77e2700 (LWP 28344) thread3 (d=0x0) at thread_bug.c:11
2 Thread 0x7ffff7fe3700 (LWP 28303) 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 28260) 0x000000362f8725db in _IO_new_file_overflow () from /lib64/libc.so.6
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff7fe3700 (LWP 28303))]#0 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
(gdb) next
Single stepping until exit from function nanosleep,
which has no line number information.
[Switching to Thread 0x7ffff77e2700 (LWP 28344)]

Breakpoint 1, thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff7fe3700 (LWP 28303))]#0 0x000000362f8acce9 in nanosleep () from /lib64/libc.so.6
(gdb) next
Single stepping until exit from function nanosleep,
which has no line number information.
0x000000362f8acb50 in sleep () from /lib64/libc.so.6
(gdb) info threads
3 Thread 0x7ffff77e2700 (LWP 28344) thread3 (d=0x0) at thread_bug.c:11
* 2 Thread 0x7ffff7fe3700 (LWP 28303) 0x000000362f8acb50 in sleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 28260) 0x000000362f8476f0 in vfprintf () from /lib64/libc.so.6
(gdb) thread 3
[Switching to thread 3 (Thread 0x7ffff77e2700 (LWP 28344))]#0 thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) next
12 l = c/2*2; /* should have been l = c/(2*2); */
(gdb) watch (k >= 2)
No symbol "k" in current context.
(gdb) watch (l >= 2)
Hardware watchpoint 4: (l >= 2)
(gdb) continue
Continuing.
Thread 1: 0
Thread 1: 1
Thread 1: 2
Thread 1: 3
Thread 1: 4
[...]
Hardware watchpoint 4: (l >= 2)

Old value = 0
New value = 1
thread3 (d=0x0) at thread_bug.c:13
13 w[l] = c;
(gdb) print l
$1 = 2
(gdb) print c
$2 = 2
(gdb) quit
</pre>
= Disassembling =
[[Category:debugger software]][[Category:bwUniCluster]]

File:Map mainwin overview.jpg

2016-01-20T14:53:30Z

R Doros: Allinea map tool main window time bars

Allinea map tool main window time bars

File:Map mainwin source.jpg

2016-01-20T11:58:09Z

R Doros: Allinea map tool main window source code view

Allinea map tool main window source code view

File:Map mainwin selection source.jpg

2016-01-20T11:57:28Z

R Doros: Allinea map tool main window source code view (with selected part)

Allinea map tool main window source code view (with selected part)

File:Map mainwin selection overview.jpg

2016-01-20T11:56:23Z

R Doros: Allinea map tool main window (with selected part)

Allinea map tool main window (with selected part)

File:Map mainwin selection function.jpg

2016-01-20T11:55:13Z

R Doros: Allinea map tool main window parallel stack view (selected part)

Allinea map tool main window parallel stack view (selected part)

File:Map mainwin presets.jpg

2016-01-20T11:53:39Z

R Doros: Allinea map tool main window presets overview

Allinea map tool main window presets overview

File:Map mainwin output.jpg

2016-01-20T11:52:41Z

R Doros: Allinea map tool main window tool output overview

Allinea map tool main window tool output overview

File:Map mainwin mpi.jpg

2016-01-20T11:51:26Z

R Doros: Allinea map tool main window mpi overview

Allinea map tool main window mpi overview

File:Map mainwin memory.jpg

2016-01-20T11:50:12Z

R Doros: Allinea map tool main window memory overview

Allinea map tool main window memory overview

File:Map mainwin ins.jpg

2016-01-20T11:49:21Z

R Doros: Allinea map tool main window instruction overview

Allinea map tool main window instruction overview

File:Map mainwin function.jpg

2016-01-20T11:47:48Z

R Doros: Allinea map tool main window parallel stack view

Allinea map tool main window parallel stack view

File:Map mainwin disk.jpg

2016-01-20T11:46:46Z

R Doros: Allinea map tool main window disc overview

Allinea map tool main window disc overview

File:Map mainwin cpu.jpg

2016-01-20T11:44:39Z

R Doros: Allinea map tool main window cpu overview

Allinea map tool main window cpu overview

File:Map mainwin.png

2016-01-20T11:00:41Z

R Doros: Allinea map tool main window

Allinea map tool main window

File:Map startwin.jpg

2016-01-20T10:59:36Z

R Doros: Allinea map tool start window

Allinea map tool start window

Development/Parallel Programming

2015-07-02T08:39:09Z

R Doros: /* Usage */

{| style="border-style: solid; border-width: 1px"
! Navigation: [[BwHPC_Best_Practices_Repository|bwHPC BPR]]
|}

= Introduction =
This page will provide information regarding the supported parallel programming paradigms and specific hints on their usage.
Please refer to the [[BwUniCluster_Environment_Modules|Modules Documentation]] how to setup your environment on bwUniCluster to load a specific software installation.

= OpenMP =
== General Information ==
OpenMP is a mature specification [http://openmp.org/wp/openmp-specifications/] to allow easy, portable, and most importantly incremental node-level parallelisation of code.
Being a thread-based approach, OpenMP is aimed at more fine-grained parallelism than [[BwHPC_Best_Practices_Repository#MPI|MPI]].
Although there have been extensions to extend OpenMP for inter-node parallelisation, it is a node-level approach aimed to make best usage of a node's cores.

With regard to ease-of-use, OpenMP is ahead of any other common approach: the source-code is annotated using <tt>#pragma omp</tt> or <tt>!$omp</tt> statements, in C/C++ and Fortran respectively.
Whenever the compiler encompasses a semantic block of code encapsulated in a parallel region, this block of code is transparently compiled into a function, which is passed to a so-called team-of-threads upon entering this semantic block. This fork-join model of execution eases a lot of the programmer's pain involved with Threads.
Being a loop-centric approach, OpenMP is aimed at codes with long/time-consuming loops.
A single combined directive <tt>pragma omp parallel for</tt> will tell the compiler to automatically parallelize the ensuing for-loop.

The following example is a bit more advanced in that even reductions of variables over multiple threads are easily parallelizable:
<source lang="c">
for (int i=0, sum = 0.0; i < VECTOR_LEN; i++)
norm2 += (v[i]*v[i]);
</source>
is parallelized by just adding a single line as in:
<source lang="c">
# pragma omp parallel for reduction(+:norm2)
for (int i=0, sum = 0.0; i < VECTOR_LEN; i++)
norm2 += (v[i]*v[i]);
</source>
With <tt>VECTOR_LENGTH</tt> being large enough, this piece of code compiled with OpenMP will run in parallel, exhibiting very nice speedup.
Compiled without, the code remains as is. Developpers may therefore incrementally parallelize their application based on the profile derived from performance analysis tools, starting with the most time-consuming loops.
Using OpenMP's concise API, one may query the number of running threads, the number of processors, a time to calculate runtime, and even set parameters such as the number of threads to execute a parallel region.

The OpenMP-4.0 specification added support for the SIMD-directive to better utilize SIMD-vectorization, as well as integrating directives to offload computation to accelerators using the <tt>target</tt> directive: these are integrated into the Intel Compiler and are actively being worked on for the GNU compiler, some restrictions may apply.

== OpenMP Best Practice Guide ==
The following silly example to calculate the squared Euklidian Norm shows some techniques:
<source lang="c">
#include <stdlib.h>
#include <stdio.h>
#include <omp.h>

#define VECTOR_LENGTH 5

int main (int argc, char * argv[])
{
int len = VECTOR_LENGTH;
int i;
double * v;
double norm2 = 0.0;
double t1, tdiff;

if (argc > 1)
len = atoi (argv[1]);
v = malloc (len * sizeof(double));

t1 = omp_get_wtime();
// Initialization already with (the same number of) threads
#pragma omp parallel for
for (i=0; i < len; i++) {
v[i] = i;
}

// Now aggregate the sum-of-squares by specifying a reduction
#pragma omp parallel for reduction(+:norm2)
for(i=0; i < len; i++) {
norm2 += (v[i]*v[i]);
}
tdiff = omp_get_wtime() - t1;

printf ("norm2: %f Time:%f\n", norm2, tdiff);
return 0;
}
</source>


* Group independent parallel sections together: in the above example, You may combine those two sections into one larger parallel block. This will just once enter the parallel region (in the fork-join model) instead of twice. Especially in inner loops, this will considerably decrease overhead.
* Compile with the Intel compiler's option <tt>-diag-enable sc-parallel3</tt> to get the further warnings on thread-safety, performance, etc. The following code with loop-carried dependency will e.g. compile fine (aka without warning):
<source lang="c">
#pragma omp parallel for reduction(+:norm2)
for(i=1; i < len-1; i++) {
v[i] = v[i-1]+v[i+1];
}
</source>
However the Intel compiler with <tt>-diag-enable sc-parallel3</tt> will produce the following warning:
<tt>warning #12246: variable "v" has loop carried data dependency that may lead to incorrect program execution in parallel mode; see (file:omp_norm2.c line:32)</tt>
* Always specify <tt>default(none)</tt> on larger parallel regions in order to specifically set the visibility of variables to either <tt>shared</tt> or <tt>private</tt>.
* Try to restructure code to allow for <tt>nowait</tt>: OpenMP defines synchronization points (implied barriers) at the end of worksharing constructs such as the <tt>pragma omp for<tt> directive. If the ensuing section of code does not depend on data being generated inside the parallel section, adding the <tt>nowait</tt> clause to the worksharing directive allows the compiler to eliminate this synchronization point. This reduces overhead, allows for better overlap and better utilization of the processor's ressources. This might imply however to restructure the code (move portions of independent code in between dependent works-sharing constructs).

== Usage ==
OpenMP is supported by various compilers, here the usage for two main compilers [[BwHPC_BPG_Compiler#GCC|GCC]] and [[BwHPC_BPG_Compiler#Intel Suite|Intel Suite]] are introduced.
For both compilers, You first need to turn on OpenMP support by specifying a parameter on the compiler's command-line.
In case You make function calls to OpenMP's API, You also need to include the header-file <tt>omp.h</tt>.
OpenMP's API allows to query or set the number of threads, query the number of processors, get a wall-clock time to measure execution times, etc.

=== OpenMP with GNU Compiler Collection ===
Starting with version 4.2 the gcc compiler supports OpenMP-2.5.
Since then the analysis capabilities of the GNU compiler have steadily improved.
The installed compilers support OpenMP-3.1.


To use OpenMP with the gcc-compiler, pass <tt>-fopenmp</tt> as parameter.

=== OpenMP with Intel Compiler ===
The Intel Compiler's support for OpenMP is more advanced than gcc's -- especially in term of programmer support.
To use OpenMP with the Intel compiler, pass <tt>-openmp</tt> as command-line parameter.

One may get very insightful information about OpenMP, when compiling with
* Compiling with <tt>-openmp-report2</tt> to get information, which loops were parallelized and a reason why not.
* Compiling with <tt>-diag-enable sc-parallel3</tt> to get errors and warnings about your sources weaknesses with regard to parallelization (see example below).

= MPI =
In this section, You will find information regarding the supported installations of the Message-Passing Interface libraries and their usage.
Due to the Fortran interface ABI, all MPI-libraries are normally bound to a specific compiler-vendor and even the specific compiler version.
Therefore, as listed in [[BwHPC_BPG_Compiler]] two compilers are supported on bwUniCluster: [[BwHPC_BPG_Compiler#GCC|GCC]] and [[BwHPC_BPG_Compiler#Intel Suite|Intel Suite]].
As both compilers are continously improving, the communication libraries will be adopted in lock-step.

With a set of different implementations, there comes the problem of choice. These pages should inform the user of the communication libraries, what considerations should be done with regard to performance, maintainability and debugging -- in general tool support -- of the various implementations.

== MPI Introduction ==
The Message-Passing Interface is a standard provided by the [http://www.mpi-forum.org MPI-Forum] which regularly convenes for the [http://meetings.mpi-forum.org MPI-Forum Meetings] to update this standard. The current version is MPI-3.0 available as [http://mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf PDF].
This document defines the API of over 300 functions for the C- and the Fortran-language -- however, You will certainly not need all of them to begin with.

Every MPI-conforming program needs to call <tt>MPI_Init()</tt> and <tt>MPI_Finalize()</tt> upon start and shutdown -- or <tt>MPI_Abort()</tt> in case of an abnormal termination.
After initialization the programm may call any other MPI-function, specifically communication functions.
However to do so, it is required to find out how many processes the program has been started with, using <tt>MPI_Comm_size()</tt> and what number (here called a rank= this particular process has using <tt>MPI_Comm_rank()</tt>.
Communication is always relative to a so-called communicator -- the default one after initialization being called <tt>MPI_COMM_WORLD</tt>-

There's basically three ways of communication:
* two-sided communication using point-to-point (often abbreviated P2P) functions, such as <tt>MPI_Send()</tt> and <tt>MPI_Recv()</tt>, which always involves two participating processes,
* collectice communcation functions (often abbreviated as colls) involve multiple processes, examples are <tt>MPI_Bcast()</tt> and <tt>MPI_Reduce()</tt>,
* one-sided communication, where communication between two processes is initiated by one-process, only. With proper RMA-hardware support and careful programming, this may allow higher performance or scalibility.

All parts of the programm, which reference MPI functionality need to be compiled with the '''same''' compiler settings/include files and linked to the same MPI-Library. This is stressed here, since without taking pre-cautions, a different MPI's header may be included, resulting in funny errors: consider that Intel MPI is derived from MPIch, with MPI-datatypes being C <tt>int</tt>s, while Open MPI uses pointers to structures (the former being 4, the latter being 8 bytes on bwUniCluster).
To ease the programmer's life, MPI implementations offer compiler-wrappers, e.g. <tt>mpicc</tt> for C and <tt>mpif90</tt> for Fortran90 for compilation and linking, taking care to include all required libraries.

All programs must be started using the <tt>mpirun</tt> or <tt>mpiexec</tt> command. Depending on the actual implementation, it uses different arguments, however the following works with any MPI:
* <tt>mpirun -np 128 ./app</tt> starts 128 processes (with ranks 0 to 127)
* <tt>mpiexec -n 128 -hostfile mynodes.txt ./app</tt> starts 128 processes on only the nodes listed line-by-line in the provided text-file <tt>mynodes.txt</tt>.
* <tt>mpiexec -n 64 ./app1 : -n 64 ./app2</tt> starts 128 processes, 64 of which execute <tt>app1</tt>, the other 64 execute <tt>app2</tt>. All processes however participate in the same <tt>MPI_COMM_WORLD</tt> and therefore must accordingly take care about their respective ranks.
Please note, that process placement (e.g. a round-robin scheme), and specifically process-binding to sockets is MPI-implementation dependant.

== MPI Best Practice Guide ==
Specific performance considerations with regard to MPI (independent of the implementation):
* No communication at all is best: Only communicate between processes if at all necessary. Consider that file-access is "communication" as well.
* If communication is done with multiple processes, try to involve as many processes in just one call: MPI optimizes the communication pattern for so-called "collective communication" to take advantage of the underlying network (with regard to network topology, message sizes, queueing capabilities of the network interconnect, etc.). Therefore try to always think in collective communication, if a communication pattern involves a group of processes.
* Try to group processes together: Function calls like <tt>MPI_Cart_create</tt> will come in handy for applications with cartesian domains but also general communicators derived from <tt>MPI_COMM_WORLD</tt> using <tt>MPI_Comm_split()</tt> may benefit by MPI's knowing the underlying network topology. Use MPI3's <tt>MPI_Comm_split_type()</tt> with <tt>MPI_COMM_TYPE_SHARED</tt> for a sub-communicator with processes having access to the same shared memory region (aka on bwUniCluster the same node).
* File-accesses to load / store data '''must''' be done collectively: Writing to storage, or even reading the initialization data -- all of which involves getting data from/to all MPI processes -- must be done collectively. MPI's Parallel IO offers a rich API to read and distribute the data access -- in order to take advantage of parallel filesystems like Lustre. A many-fold performance improvement may be seen by writing data in large chunks in collective fashion -- and at the same time being nice to other users and applications.
* Try to hide the communication by computation: Try to hide (some) of the cost of communication of Point-to-point communication by using non-blocking / immediate P2P-calls (<tt>MPI_Isend</tt> and <tt>MPI_Irecv</tt> et al, followed by <tt>MPI_Wait</tt> or <tt>MPI_Test</tt> et al). This may allow the MPI-implementation to initiate or even offload communication to the network interconnect and resume executing your application, while data is being transferred. MPI-3 adds non-blocking collectives, e.g. <tt>MPI_Ibcast()</tt> or <tt>MPI_Iallreduce()</tt>. For extra credit, explain the use-cases of <tt>MPI_Ibarrier()</tt>.
* Every call to MPI may trigger an access to physical hardware -- limit it: When calling communication-related functions such as <tt>MPI_Test</tt> to check whether a specific communication has finished, the queue of the network adapter may need to be queried. This memory access or even physical hardware access to query the state will cost cycles. Therefore, the programmer should combine multiple requests with functions such as <tt>MPI_Waitall()</tt> or <tt>MPI_Waitany()</tt> or their Test*-counterparts.
* Make usage of derived datatypes: instead of manually copying data into temporary, even newly allocated memory, describe the data-layout to MPI -- and let the implementation, or even the network HCA's hardware do the data fetching.
* Bind Your processes to sockets: Operating Systems are good in making best use of the ressources -- which sometimes involves moving tasks from one core to another, or even (though more unlikely since the OS' heuristics try to avoid it) to another socket, with the obvious effects: Caches are cold, every memory access to memory allocated on the previous socket "has to travel the bus". This is particularly happening if You have multiple OpenMP parallel regions which are separated by code that does IO -- and threads are sleeping -- the processes doing IO may wander to a different socket... Bind Your processes to at least the socket. All major MPIs support this binding (see below).
* Do not use the C++ interface: First of all, it has been marked as deprecated in the MPI-3.0 standard, since it added little benefit to C++ programmers over the C-interface. Moreover, since MPI implementations are written in C, the interface adds another level of indirection and therefore a bit of overhead in terms of instructions and Cache misses.

== Open MPI ==
The [http://www.open-mpi.org Open MPI] library is an open, flexible and nevertheless performant implementation of MPI-2 and MPI-3. Licensed under BSD, it is being actively developed by an open community of industry and research institutions.
The flexibility comes in handy: using the concept of a [http://www.open-mpi.org/faq/?category=tuning#mca-def MCA] (aka a plugin) Open MPI supports many different network interconnects (Infinband, TCP, Cray, etc.) , on the other hand, a installation may be tailored to suite an installation, e.g. the network (Infiniband with specific settings), the main startup-mechanism, etc.
Furthermore, the [http://www.open-mpi.org/faq/ FAQ] offers hints on [http://www.open-mpi.org/faq/?category=tuning performance tuning].

=== Usage ===
Like other MPI implementations, after loading the [[BwUniCluster_Environment_Modules|module]], Open MPI provides the compiler-wrappers <tt>mpicc</tt>, <tt>mpicxx</tt> and <tt>mpifort</tt> (or for
versions lower than 1.7 <tt>mpif77</tt> and <tt>mpif90</tt>) for the C-, C++ and Fortran compilers respectively. Albeit their usage is not required, these wrappers are handy to not have to use the command-line options for header- or library directories, aka <tt>-I</tt> and <tt>-L</tt>, as well as the actual needed MPI-libraries itselve.

=== Further information ===

Open MPI also features a few specific functionalities that will help users and developpers, alike:
* Open MPI's tool <tt>ompi_info</tt> allows seeing all of Open MPI's installed MCA components and their specific options.
Without any option the user gets a list of flags, the Open MPI installation was compiled for (version of compilers, specific configure-flags, e.g. debugging, or profiling options). Furthermore, using <tt>ompi_info --param all all</tt> one may see all of the MCA's options, e.g. that the default PML-MCA uses an initial free-list of 4 blocks (increased by 64 upon first encountering this limit):
<tt>ompi_info --param ob1 all</tt> -- which may be increased for applications that are certain to benefit from a larger value upon startup.
* Open MPI allows adapting MCA parameters on the command-line: parameters may be supplied, e.g. the above-mentioned parameter <tt>mpirun -np 128 --mca mpirun -np 16 --mca pml_ob1_free_list_num 128 ./mpi_stub</tt>.
* Open MPI internally uses the tool [http://www.open-mpi.org/projects/hwloc/ hwloc] for node-local processor-information, as well as process- and memory-affinity. This tool also is a good tool to get information on the node's processor topology and Cache-information. This may be used to optimize and balance memory usage or for choosing a better ratio of MPI processes per node vs. OpenMP threads per core.

Development/GCC

2015-04-23T06:16:00Z

R Doros:

{|{{Softwarebox}}
|-
! colspan="2" style="text-align:center" | GCC
|-
| module load
| compiler/gnu/xxx
|-
| License
| GPL
|-
|Citing
|
|-
| Links
| [http://gcc.gnu.org/ Homepage]
|
|-
| Graphical Interface
| No
|-
| Included in module
| gcc, g++, gfortran
|}
The '''GNU Compiler Collection (GCC)''' consists of tools to compile C, C++ and Fortran programs:
{| border="1" style="margin:5px;border-collapse:collapse"
|-
|style="padding:3px"| gcc
|style="padding:3px"| GNU C compiler
|-
|style="padding:3px"| g++
|style="padding:3px"| GNU C++ compiler
|-
|style="padding:3px"| gfortran
|style="padding:3px"| GNU Fortran compiler
|}

'''Loading:''' There is a version of GCC available on the system without loading a module but it may be outdated and it is recommended to load the GNU compiler module. To get a list of all the different versions installed on the system execute the following command:
<pre>$ module avail compiler/gnu</pre>
There is a default version which will be loaded when no version is explicitly specified, so the command
<pre>$ module load compiler/gnu</pre>
will load the default version.

'''Online documentation:''' http://gcc.gnu.org/onlinedocs/

'''Local documentation:''' For version specific documentation see the help page of the module. For example
<pre>$ module help compiler/gnu</pre>
will show the information for the default version.
For detailed lists of the different program options consult the particular man page:
<pre>
$ man gcc
$ man g++
$ man gfortran
</pre>

'''Optimizations:''' You can turn on various optimization options to enhance the performance of your program. Which options are the best depends on the specific program and can be determined by benchmarking your code. A command which gives good performance and a decent file size is
<pre>$ gcc -march=native -O2 ex.c -o ex</pre>
There are more aggressive optimization flags but the compiled programs can get quite large and the compilation process will probably take much longer. Moreover it can happen that the so compiled program is even slower. Such a command would be for example
<pre>$ gcc -march=native -O3 ex.c -o ex</pre>
For a complete list of all the optimization options execute
<pre>$ gcc --help=optimizers</pre>

'''Profiling with gprof:''' Gprof is the profiler which belongs to the gcc compiler. Gprof is also installed on the system and can be used to profile your code. The profiler supports C, C++, Pascal and Fortran 77 program code. With this tool it is possible to analyse call times and time spent within program functions.

The first required step is to compile your program with the profiling flag "-pg".
<pre>$ gcc -pg ex.c -o ex</pre>
Compiled in this way your program will generate profiling data during execution. By default a file named gmon.out can be found in the work directory after running the program.
<pre>$ ./ex</pre>
The next step is to run the gprof program to analyse the gmon.out profiling data file. This file contains profiling data concerning the program execution, like an overview, time information or the call graph, in human readable format.
<pre>$ gprof gmon.out ex > outputfile.txt</pre>

[[Category:Compiler_software]][[Category:bwUniCluster]]

Development/GDB

2014-05-21T09:48:10Z

R Doros: mt session finished

{|{{Softwarebox}}
|-
! colspan="2" style="text-align:center" | GDB: The GNU Project Debugger
|-
| module load
| devel/gdb

|-
| License
| [http://www.gnu.org/licenses/ GPL]
|-
|Citing
|
|-
| Links
| [http://www.gnu.org/software/gdb/ Homepage]; [http://www.gnu.org/software/gdb/documentation/ Documentation]; [https://sourceware.org/gdb/wiki/ Wiki]; [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists]
|
|-
| Graphical Interface
| No 
|-
| Included in module
| 
|}

The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.

== Loading ==
It is possible to choose between several versions of GDB and GCC. By default both programms are installed. You can check the version which is currently used with:
<pre>$ gdb --version</pre>
<pre>$ gcc --version</pre>

To use a different version of GCC or GDB, you have to load it through the module system. For example you need GDB 7.7 or higher for GCC version 4.8 and above. In this case load the devel/gdb module.

How to check which versions are available:
<pre>$ module avail</pre>

How to load the desired version (e.g. GDB version 7.7):
<pre>$ module load devel/gdb/7.7</pre>

If you want to use IDB load the Intel compiler module:
<pre>$ module load compiler/intel</pre>

== Documentation ==
For '''online documentation''' see the links section in the summary table at the top of this page. For '''local documentation''' consult the man page.
<pre>$ man gdb</pre>
or
<pre>$ man idb</pre>

== Basic commands ==
The code you want to debug should be compiled with the -g option. If the optimization flag is not set, the GCC will still do some basic optimization. Therefore, it is recommended to turn off the optmization explicitly with the -O0 parameter for debugging. To start a debug session for a program execute GDB with the program path as parameter:
<pre>$ gdb ./example</pre>

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

{| {{Table}}
|-
! Command
! Description
|-
| help ''cmd''
| Show help for command ''cmd''.
|-
| break ''func''
| Set a breakpoint at function ''func''.
|-
| run
| Start program.
|-
| next
| Go to next program line. Do not enter functions.
|-
| step
| Go to next program line. Enter functions.
|-
| list
| Show the surrounding source code of the currently processed line.
|-
| print ''expr''
| Print the value of the expression ''expr''.
|-
| display ''expr''
| Display the value of the expression ''expr'' every time the program stops.
|-
| watch ''expr''
| Stop when value of the expression ''expr'' changes.
|-
| continue
| Continue execution until a breakpoint or a watchpoint appears.
|-
| backtrace
| Print a list of functions that are currently active.
|-
| quit
| Exit GDB.
|}

'''Example:'''
We debug the following program called bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>

int global = 0;

void begin() {
global = 1;
}

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}

void end() {
global = 2;
}

int main() {
begin();
loop();
end();

return 0;
}
</source>

'''Sample GDB session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27 loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
(gdb) next
27 loop();
(gdb) step
loop () at bug.c:13
13 for(i = 0; i < 8; i++)
(gdb) next
15 k = i/2*2;
(gdb) next
16 v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16 v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit
</pre>

== Core dumps ==
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
<pre>$ ulimit -c unlimited</pre>
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
<pre>$ gdb ./ex core.xxx</pre>
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.

== Multithreaded debugging ==
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step.
The following commands are useful for multithreaded debugging:

{| {{Table}}
|-
! Command
! Description
|-
| info threads
| Shows the status of all existing threads.
|-
| thread ''num''
| Switches to the thread with the number ''num''
|}

'''Example:'''
We debug the following program called thread_bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>
#include <pthread.h>

pthread_t thread;

void* thread3 (void* d)
{
int w[2];
int c, l;

for(c = 0; c < 8; c++) {
l = c/2*2; /* should have been l = c/(2*2); */
w[l] = c;
}

return NULL;
}

void* thread2 (void* d)
{
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
sleep(4);
k = i/(2*2); /* should have been k = i/(2*2); */
v[k] = i;
}

return NULL;
}

int main (){

pthread_create (&thread, NULL, thread2, NULL);
pthread_create (&thread, NULL, thread3, NULL);

//Thread 1
int count1 = 0;

while(count1 < 4000){
printf("Thread 1: %d\n", count1++);
}

pthread_join(thread, NULL);
return 0;
}
</source>

'''Sample GDB thread session:'''
<pre>
$ gcc -g thread_bug.c -o thread_bug -lpthread
$ gdb ./thread_bug
[...]
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break thread3
Breakpoint 1 at 0x40060c: file thread_bug.c, line 11.
(gdb) break thread2
Breakpoint 2 at 0x400650: file thread_bug.c, line 24.
(gdb) break main
Breakpoint 3 at 0x40069e: file thread_bug.c, line 35.
(gdb) run
Starting program: /tank/home/doros/.t/thread_bug
[Thread debugging using libthread_db enabled]

Breakpoint 3, main () at thread_bug.c:35
35 pthread_create (&thread, NULL, thread2, NULL);
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64
(gdb) info threads
* 1 Thread 0x7ffff7fe5700 (LWP 28260) main () at thread_bug.c:35
(gdb) next
[New Thread 0x7ffff7fe3700 (LWP 28303)]
36 pthread_create (&thread, NULL, thread3, NULL);
(gdb) info threads
2 Thread 0x7ffff7fe3700 (LWP 28303) thread2 (d=0x0) at thread_bug.c:24
* 1 Thread 0x7ffff7fe5700 (LWP 28260) main () at thread_bug.c:36
(gdb) next
[Switching to Thread 0x7ffff7fe3700 (LWP 28303)]

Breakpoint 2, thread2 (d=0x0) at thread_bug.c:24
24 for(i = 0; i < 8; i++) {
(gdb) next
25 sleep(4);
(gdb) next
[New Thread 0x7ffff77e2700 (LWP 28344)]
[Switching to Thread 0x7ffff77e2700 (LWP 28344)]

Breakpoint 1, thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) info threads
* 3 Thread 0x7ffff77e2700 (LWP 28344) thread3 (d=0x0) at thread_bug.c:11
2 Thread 0x7ffff7fe3700 (LWP 28303) 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 28260) 0x000000362f8725db in _IO_new_file_overflow () from /lib64/libc.so.6
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff7fe3700 (LWP 28303))]#0 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
(gdb) next
Single stepping until exit from function nanosleep,
which has no line number information.
[Switching to Thread 0x7ffff77e2700 (LWP 28344)]

Breakpoint 1, thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff7fe3700 (LWP 28303))]#0 0x000000362f8acce9 in nanosleep () from /lib64/libc.so.6
(gdb) next
Single stepping until exit from function nanosleep,
which has no line number information.
0x000000362f8acb50 in sleep () from /lib64/libc.so.6
(gdb) info threads
3 Thread 0x7ffff77e2700 (LWP 28344) thread3 (d=0x0) at thread_bug.c:11
* 2 Thread 0x7ffff7fe3700 (LWP 28303) 0x000000362f8acb50 in sleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 28260) 0x000000362f8476f0 in vfprintf () from /lib64/libc.so.6
(gdb) thread 3
[Switching to thread 3 (Thread 0x7ffff77e2700 (LWP 28344))]#0 thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) next
12 l = c/2*2; /* should have been l = c/(2*2); */
(gdb) watch (k >= 2)
No symbol "k" in current context.
(gdb) watch (l >= 2)
Hardware watchpoint 4: (l >= 2)
(gdb) continue
Continuing.
Thread 1: 0
Thread 1: 1
Thread 1: 2
Thread 1: 3
Thread 1: 4
[...]
Hardware watchpoint 4: (l >= 2)

Old value = 0
New value = 1
thread3 (d=0x0) at thread_bug.c:13
13 w[l] = c;
(gdb) print l
$1 = 2
(gdb) print c
$2 = 2
(gdb) quit
</pre>

[[Category:debugger software]]

Development/GDB

2014-05-21T09:34:43Z

R Doros: Continue mt session

{|{{Softwarebox}}
|-
! colspan="2" style="text-align:center" | GDB: The GNU Project Debugger
|-
| module load
| devel/gdb

|-
| License
| [http://www.gnu.org/licenses/ GPL]
|-
|Citing
|
|-
| Links
| [http://www.gnu.org/software/gdb/ Homepage]; [http://www.gnu.org/software/gdb/documentation/ Documentation]; [https://sourceware.org/gdb/wiki/ Wiki]; [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists]
|
|-
| Graphical Interface
| No 
|-
| Included in module
| 
|}

The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.

== Loading ==
It is possible to choose between several versions of GDB and GCC. By default both programms are installed. You can check the version which is currently used with:
<pre>$ gdb --version</pre>
<pre>$ gcc --version</pre>

To use a different version of GCC or GDB, you have to load it through the module system. For example you need GDB 7.7 or higher for GCC version 4.8 and above. In this case load the devel/gdb module.

How to check which versions are available:
<pre>$ module avail</pre>

How to load the desired version (e.g. GDB version 7.7):
<pre>$ module load devel/gdb/7.7</pre>

If you want to use IDB load the Intel compiler module:
<pre>$ module load compiler/intel</pre>

== Documentation ==
For '''online documentation''' see the links section in the summary table at the top of this page. For '''local documentation''' consult the man page.
<pre>$ man gdb</pre>
or
<pre>$ man idb</pre>

== Basic commands ==
The code you want to debug should be compiled with the -g option. If the optimization flag is not set, the GCC will still do some basic optimization. Therefore, it is recommended to turn off the optmization explicitly with the -O0 parameter for debugging. To start a debug session for a program execute GDB with the program path as parameter:
<pre>$ gdb ./example</pre>

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

{| {{Table}}
|-
! Command
! Description
|-
| help ''cmd''
| Show help for command ''cmd''.
|-
| break ''func''
| Set a breakpoint at function ''func''.
|-
| run
| Start program.
|-
| next
| Go to next program line. Do not enter functions.
|-
| step
| Go to next program line. Enter functions.
|-
| list
| Show the surrounding source code of the currently processed line.
|-
| print ''expr''
| Print the value of the expression ''expr''.
|-
| display ''expr''
| Display the value of the expression ''expr'' every time the program stops.
|-
| watch ''expr''
| Stop when value of the expression ''expr'' changes.
|-
| continue
| Continue execution until a breakpoint or a watchpoint appears.
|-
| backtrace
| Print a list of functions that are currently active.
|-
| quit
| Exit GDB.
|}

'''Example:'''
We debug the following program called bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>

int global = 0;

void begin() {
global = 1;
}

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}

void end() {
global = 2;
}

int main() {
begin();
loop();
end();

return 0;
}
</source>

'''Sample GDB session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27 loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
(gdb) next
27 loop();
(gdb) step
loop () at bug.c:13
13 for(i = 0; i < 8; i++)
(gdb) next
15 k = i/2*2;
(gdb) next
16 v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16 v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit
</pre>

== Core dumps ==
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
<pre>$ ulimit -c unlimited</pre>
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
<pre>$ gdb ./ex core.xxx</pre>
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.

== Multithreaded debugging ==
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step.
The following commands are useful for multithreaded debugging:

{| {{Table}}
|-
! Command
! Description
|-
| info threads
| Shows the status of all existing threads.
|-
| thread ''num''
| Switches to the thread with the number ''num''
|}

'''Example:'''
We debug the following program called thread_bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>
#include <pthread.h>

pthread_t thread;

void* thread3 (void* d)
{
int w[2];
int c, l;

for(c = 0; c < 8; c++) {
l = c/2*2; /* should have been l = c/(2*2); */
w[l] = c;
}

return NULL;
}

void* thread2 (void* d)
{
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
sleep(4);
k = i/(2*2); /* should have been k = i/(2*2); */
v[k] = i;
}

return NULL;
}

int main (){

pthread_create (&thread, NULL, thread2, NULL);
pthread_create (&thread, NULL, thread3, NULL);

//Thread 1
int count1 = 0;

while(count1 < 4000){
printf("Thread 1: %d\n", count1++);
}

pthread_join(thread, NULL);
return 0;
}
</source>

'''Sample GDB thread session:'''
<pre>
$ gcc -g thread_bug.c -o thread_bug -lpthread
$ gdb ./thread_bug
[...]
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break thread3
Breakpoint 1 at 0x40060c: file thread_bug.c, line 11.
(gdb) break thread2
Breakpoint 2 at 0x400650: file thread_bug.c, line 24.
(gdb) break main
Breakpoint 3 at 0x40069e: file thread_bug.c, line 35.
(gdb) run
Starting program: /tank/home/doros/.t/thread_bug
[Thread debugging using libthread_db enabled]

Breakpoint 3, main () at thread_bug.c:35
35 pthread_create (&thread, NULL, thread2, NULL);
(gdb) info threads
* 1 Thread 0x7ffff7fe5700 (LWP 49274) main () at thread_bug.c:35
(gdb) cont
Continuing.
[New Thread 0x7ffff7fe3700 (LWP 19724)]
[Switching to Thread 0x7ffff7fe3700 (LWP 19724)]

Breakpoint 2, thread2 (d=0x0) at thread_bug.c:24
24 for(i = 0; i < 8; i++) {
(gdb) info threads
* 2 Thread 0x7ffff7fe3700 (LWP 19724) thread2 (d=0x0) at thread_bug.c:24
1 Thread 0x7ffff7fe5700 (LWP 49274) 0x000000362f8e55a7 in mprotect () from /lib64/libc.so.6
(gdb) next
25 sleep(4);
(gdb) next
[New Thread 0x7ffff77e2700 (LWP 19780)]
Thread 1: 0
Thread 1: 1
[Switching to Thread 0x7ffff77e2700 (LWP 19780)]

Breakpoint 1, thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) info threads
* 3 Thread 0x7ffff77e2700 (LWP 19780) thread3 (d=0x0) at thread_bug.c:11
2 Thread 0x7ffff7fe3700 (LWP 19724) 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 49274) 0x000000362f8db7ad in write () from /lib64/libc.so.6
(gdb) next
12 l = c/2*2; /* should have been l = c/(2*2); */
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff7fe3700 (LWP 19724))]#0 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
(gdb) next
Single stepping until exit from function nanosleep,
which has no line number information.
Thread 1: 2
Thread 1: 3
Thread 1: 4
Thread 1: 5
Thread 1: 6
Thread 1: 7
Thread 1: 8
Thread 1: 9
Thread 1: 10
Thread 1: 11
Thread 1: 12
Thread 1: 13
0x000000362f8acb50 in sleep () from /lib64/libc.so.6
(gdb) thread 3
[Switching to thread 3 (Thread 0x7ffff77e2700 (LWP 19780))]#0 0x0000003600000007 in ?? ()
(gdb) info threads
* 3 Thread 0x7ffff77e2700 (LWP 19780) 0x0000003600000007 in ?? ()
2 Thread 0x7ffff7fe3700 (LWP 19724) 0x000000362f8acb50 in sleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 49274) 0x000000362f8db7ad in write () from /lib64/libc.so.6

</pre>

[[Category:debugger software]]

Development/GDB

2014-05-20T14:50:29Z

R Doros: minor edit on session

{|{{Softwarebox}}
|-
! colspan="2" style="text-align:center" | GDB: The GNU Project Debugger
|-
| module load
| devel/gdb

|-
| License
| [http://www.gnu.org/licenses/ GPL]
|-
|Citing
|
|-
| Links
| [http://www.gnu.org/software/gdb/ Homepage]; [http://www.gnu.org/software/gdb/documentation/ Documentation]; [https://sourceware.org/gdb/wiki/ Wiki]; [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists]
|
|-
| Graphical Interface
| No 
|-
| Included in module
| 
|}

The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.

== Loading ==
It is possible to choose between several versions of GDB and GCC. By default both programms are installed. You can check the version which is currently used with:
<pre>$ gdb --version</pre>
<pre>$ gcc --version</pre>

To use a different version of GCC or GDB, you have to load it through the module system. For example you need GDB 7.7 or higher for GCC version 4.8 and above. In this case load the devel/gdb module.

How to check which versions are available:
<pre>$ module avail</pre>

How to load the desired version (e.g. GDB version 7.7):
<pre>$ module load devel/gdb/7.7</pre>

If you want to use IDB load the Intel compiler module:
<pre>$ module load compiler/intel</pre>

== Documentation ==
For '''online documentation''' see the links section in the summary table at the top of this page. For '''local documentation''' consult the man page.
<pre>$ man gdb</pre>
or
<pre>$ man idb</pre>

== Basic commands ==
The code you want to debug should be compiled with the -g option. If the optimization flag is not set, the GCC will still do some basic optimization. Therefore, it is recommended to turn off the optmization explicitly with the -O0 parameter for debugging. To start a debug session for a program execute GDB with the program path as parameter:
<pre>$ gdb ./example</pre>

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

{| {{Table}}
|-
! Command
! Description
|-
| help ''cmd''
| Show help for command ''cmd''.
|-
| break ''func''
| Set a breakpoint at function ''func''.
|-
| run
| Start program.
|-
| next
| Go to next program line. Do not enter functions.
|-
| step
| Go to next program line. Enter functions.
|-
| list
| Show the surrounding source code of the currently processed line.
|-
| print ''expr''
| Print the value of the expression ''expr''.
|-
| display ''expr''
| Display the value of the expression ''expr'' every time the program stops.
|-
| watch ''expr''
| Stop when value of the expression ''expr'' changes.
|-
| continue
| Continue execution until a breakpoint or a watchpoint appears.
|-
| backtrace
| Print a list of functions that are currently active.
|-
| quit
| Exit GDB.
|}

'''Example:'''
We debug the following program called bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>

int global = 0;

void begin() {
global = 1;
}

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}

void end() {
global = 2;
}

int main() {
begin();
loop();
end();

return 0;
}
</source>

'''Sample GDB session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27 loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
(gdb) next
27 loop();
(gdb) step
loop () at bug.c:13
13 for(i = 0; i < 8; i++)
(gdb) next
15 k = i/2*2;
(gdb) next
16 v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16 v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit
</pre>

== Core dumps ==
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
<pre>$ ulimit -c unlimited</pre>
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
<pre>$ gdb ./ex core.xxx</pre>
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.

== Multithreaded debugging ==
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step.
The following commands are useful for multithreaded debugging:

{| {{Table}}
|-
! Command
! Description
|-
| info threads
| Shows the status of all existing threads.
|-
| thread ''num''
| Switches to the thread with the number ''num''
|}

'''Example:'''
We debug the following program called thread_bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>
#include <pthread.h>

pthread_t thread;

void* thread3 (void* d)
{
int w[2];
int c, l;

for(c = 0; c < 8; c++) {
l = c/2*2; /* should have been l = c/(2*2); */
w[l] = c;
}

return NULL;
}

void* thread2 (void* d)
{
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
sleep(4);
k = i/(2*2); /* should have been k = i/(2*2); */
v[k] = i;
}

return NULL;
}

int main (){

pthread_create (&thread, NULL, thread2, NULL);
pthread_create (&thread, NULL, thread3, NULL);

//Thread 1
int count1 = 0;

while(count1 < 4000){
printf("Thread 1: %d\n", count1++);
}

pthread_join(thread, NULL);
return 0;
}
</source>

'''Sample GDB thread session:'''
<pre>
$ gcc -g thread_bug.c -o thread_bug -lpthread
$ gdb ./thread_bug
[...]
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break thread3
Breakpoint 1 at 0x40060c: file thread_bug.c, line 11.
(gdb) break thread2
Breakpoint 2 at 0x400650: file thread_bug.c, line 24.
(gdb) break main
Breakpoint 3 at 0x40069e: file thread_bug.c, line 35.
(gdb) run
Starting program: /tank/home/doros/.t/thread_bug
[Thread debugging using libthread_db enabled]

Breakpoint 3, main () at thread_bug.c:35
35 pthread_create (&thread, NULL, thread2, NULL);
(gdb) info threads
* 1 Thread 0x7ffff7fe5700 (LWP 49049) main () at thread_bug.c:35
(gdb) next
[New Thread 0x7ffff7fe3700 (LWP 49077)]
36 pthread_create (&thread, NULL, thread3, NULL);
(gdb) info threads
2 Thread 0x7ffff7fe3700 (LWP 49077) thread2 (d=0x0) at thread_bug.c:24
* 1 Thread 0x7ffff7fe5700 (LWP 49049) main () at thread_bug.c:36
(gdb) next
[Switching to Thread 0x7ffff7fe3700 (LWP 49077)]

Breakpoint 2, thread2 (d=0x0) at thread_bug.c:24
24 for(i = 0; i < 8; i++) {
(gdb) info threads
* 2 Thread 0x7ffff7fe3700 (LWP 49077) thread2 (d=0x0) at thread_bug.c:24
1 Thread 0x7ffff7fe5700 (LWP 49049) 0x000000362f8e55a7 in mprotect () from /lib64/libc.so.6
(gdb) next
25 sleep(4);
(gdb) next
[New Thread 0x7ffff77e2700 (LWP 49086)]
[Switching to Thread 0x7ffff77e2700 (LWP 49086)]

Breakpoint 1, thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) info threads
* 3 Thread 0x7ffff77e2700 (LWP 49086) thread3 (d=0x0) at thread_bug.c:11
2 Thread 0x7ffff7fe3700 (LWP 49077) 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 49049) 0x000000362f874720 in _IO_setb_internal () from /lib64/libc.so.6
(gdb) next
12 l = c/(2*2); /* should have been l = c/(2*2); */
(gdb) next
13 w[l] = c;
(gdb) next
11 for(c = 0; c < 8; c++) {
</pre>

[[Category:debugger software]]

Development/GDB

2014-05-20T14:49:15Z

R Doros: minor edit on session

{|{{Softwarebox}}
|-
! colspan="2" style="text-align:center" | GDB: The GNU Project Debugger
|-
| module load
| devel/gdb

|-
| License
| [http://www.gnu.org/licenses/ GPL]
|-
|Citing
|
|-
| Links
| [http://www.gnu.org/software/gdb/ Homepage]; [http://www.gnu.org/software/gdb/documentation/ Documentation]; [https://sourceware.org/gdb/wiki/ Wiki]; [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists]
|
|-
| Graphical Interface
| No 
|-
| Included in module
| 
|}

The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.

== Loading ==
It is possible to choose between several versions of GDB and GCC. By default both programms are installed. You can check the version which is currently used with:
<pre>$ gdb --version</pre>
<pre>$ gcc --version</pre>

To use a different version of GCC or GDB, you have to load it through the module system. For example you need GDB 7.7 or higher for GCC version 4.8 and above. In this case load the devel/gdb module.

How to check which versions are available:
<pre>$ module avail</pre>

How to load the desired version (e.g. GDB version 7.7):
<pre>$ module load devel/gdb/7.7</pre>

If you want to use IDB load the Intel compiler module:
<pre>$ module load compiler/intel</pre>

== Documentation ==
For '''online documentation''' see the links section in the summary table at the top of this page. For '''local documentation''' consult the man page.
<pre>$ man gdb</pre>
or
<pre>$ man idb</pre>

== Basic commands ==
The code you want to debug should be compiled with the -g option. If the optimization flag is not set, the GCC will still do some basic optimization. Therefore, it is recommended to turn off the optmization explicitly with the -O0 parameter for debugging. To start a debug session for a program execute GDB with the program path as parameter:
<pre>$ gdb ./example</pre>

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

{| {{Table}}
|-
! Command
! Description
|-
| help ''cmd''
| Show help for command ''cmd''.
|-
| break ''func''
| Set a breakpoint at function ''func''.
|-
| run
| Start program.
|-
| next
| Go to next program line. Do not enter functions.
|-
| step
| Go to next program line. Enter functions.
|-
| list
| Show the surrounding source code of the currently processed line.
|-
| print ''expr''
| Print the value of the expression ''expr''.
|-
| display ''expr''
| Display the value of the expression ''expr'' every time the program stops.
|-
| watch ''expr''
| Stop when value of the expression ''expr'' changes.
|-
| continue
| Continue execution until a breakpoint or a watchpoint appears.
|-
| backtrace
| Print a list of functions that are currently active.
|-
| quit
| Exit GDB.
|}

'''Example:'''
We debug the following program called bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>

int global = 0;

void begin() {
global = 1;
}

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}

void end() {
global = 2;
}

int main() {
begin();
loop();
end();

return 0;
}
</source>

'''Sample GDB session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27 loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
(gdb) next
27 loop();
(gdb) step
loop () at bug.c:13
13 for(i = 0; i < 8; i++)
(gdb) next
15 k = i/2*2;
(gdb) next
16 v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16 v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit
</pre>

== Core dumps ==
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
<pre>$ ulimit -c unlimited</pre>
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
<pre>$ gdb ./ex core.xxx</pre>
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.

== Multithreaded debugging ==
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step.
The following commands are useful for multithreaded debugging:

{| {{Table}}
|-
! Command
! Description
|-
| info threads
| Shows the status of all existing threads.
|-
| thread ''num''
| Switches to the thread with the number ''num''
|}

'''Example:'''
We debug the following program called thread_bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>
#include <pthread.h>

pthread_t thread;

void* thread3 (void* d)
{
int w[2];
int c, l;

for(c = 0; c < 8; c++) {
l = c/2*2; /* should have been l = c/(2*2); */
w[l] = c;
}

return NULL;
}

void* thread2 (void* d)
{
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
sleep(4);
k = i/(2*2); /* should have been k = i/(2*2); */
v[k] = i;
}

return NULL;
}

int main (){

pthread_create (&thread, NULL, thread2, NULL);
pthread_create (&thread, NULL, thread3, NULL);

//Thread 1
int count1 = 0;

while(count1 < 4000){
printf("Thread 1: %d\n", count1++);
}

pthread_join(thread, NULL);
return 0;
}
</source>

'''Sample GDB thread session:'''
<pre>
$ gcc -g thread_bug.c -o thread_bug -lpthread
$ gdb ./thread_bug
[...]
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break thread3
Breakpoint 1 at 0x40060c: file thread_bug.c, line 11.
(gdb) break thread2
Breakpoint 2 at 0x400650: file thread_bug.c, line 24.
(gdb) break main
Breakpoint 3 at 0x40069e: file thread_bug.c, line 35.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/thread_bug

Breakpoint 3, main () at thread_bug.c:35
35 pthread_create (&thread, NULL, thread2, NULL);
(gdb) info threads
* 1 Thread 0x7ffff7fe5700 (LWP 49049) main () at thread_bug.c:35
(gdb) next
[New Thread 0x7ffff7fe3700 (LWP 49077)]
36 pthread_create (&thread, NULL, thread3, NULL);
(gdb) info threads
2 Thread 0x7ffff7fe3700 (LWP 49077) thread2 (d=0x0) at thread_bug.c:24
* 1 Thread 0x7ffff7fe5700 (LWP 49049) main () at thread_bug.c:36
(gdb) next
[Switching to Thread 0x7ffff7fe3700 (LWP 49077)]

Breakpoint 2, thread2 (d=0x0) at thread_bug.c:24
24 for(i = 0; i < 8; i++) {
(gdb) info threads
* 2 Thread 0x7ffff7fe3700 (LWP 49077) thread2 (d=0x0) at thread_bug.c:24
1 Thread 0x7ffff7fe5700 (LWP 49049) 0x000000362f8e55a7 in mprotect () from /lib64/libc.so.6
(gdb) next
25 sleep(4);
(gdb) next
[New Thread 0x7ffff77e2700 (LWP 49086)]
[Switching to Thread 0x7ffff77e2700 (LWP 49086)]

Breakpoint 1, thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) info threads
* 3 Thread 0x7ffff77e2700 (LWP 49086) thread3 (d=0x0) at thread_bug.c:11
2 Thread 0x7ffff7fe3700 (LWP 49077) 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 49049) 0x000000362f874720 in _IO_setb_internal () from /lib64/libc.so.6
(gdb) next
12 l = c/(2*2); /* should have been l = c/(2*2); */
(gdb) next
13 w[l] = c;
(gdb) next
11 for(c = 0; c < 8; c++) {
</pre>

[[Category:debugger software]]

Development/GDB

2014-05-20T14:47:13Z

R Doros: Sourcecode changed

{|{{Softwarebox}}
|-
! colspan="2" style="text-align:center" | GDB: The GNU Project Debugger
|-
| module load
| devel/gdb

|-
| License
| [http://www.gnu.org/licenses/ GPL]
|-
|Citing
|
|-
| Links
| [http://www.gnu.org/software/gdb/ Homepage]; [http://www.gnu.org/software/gdb/documentation/ Documentation]; [https://sourceware.org/gdb/wiki/ Wiki]; [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists]
|
|-
| Graphical Interface
| No 
|-
| Included in module
| 
|}

The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.

== Loading ==
It is possible to choose between several versions of GDB and GCC. By default both programms are installed. You can check the version which is currently used with:
<pre>$ gdb --version</pre>
<pre>$ gcc --version</pre>

To use a different version of GCC or GDB, you have to load it through the module system. For example you need GDB 7.7 or higher for GCC version 4.8 and above. In this case load the devel/gdb module.

How to check which versions are available:
<pre>$ module avail</pre>

How to load the desired version (e.g. GDB version 7.7):
<pre>$ module load devel/gdb/7.7</pre>

If you want to use IDB load the Intel compiler module:
<pre>$ module load compiler/intel</pre>

== Documentation ==
For '''online documentation''' see the links section in the summary table at the top of this page. For '''local documentation''' consult the man page.
<pre>$ man gdb</pre>
or
<pre>$ man idb</pre>

== Basic commands ==
The code you want to debug should be compiled with the -g option. If the optimization flag is not set, the GCC will still do some basic optimization. Therefore, it is recommended to turn off the optmization explicitly with the -O0 parameter for debugging. To start a debug session for a program execute GDB with the program path as parameter:
<pre>$ gdb ./example</pre>

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

{| {{Table}}
|-
! Command
! Description
|-
| help ''cmd''
| Show help for command ''cmd''.
|-
| break ''func''
| Set a breakpoint at function ''func''.
|-
| run
| Start program.
|-
| next
| Go to next program line. Do not enter functions.
|-
| step
| Go to next program line. Enter functions.
|-
| list
| Show the surrounding source code of the currently processed line.
|-
| print ''expr''
| Print the value of the expression ''expr''.
|-
| display ''expr''
| Display the value of the expression ''expr'' every time the program stops.
|-
| watch ''expr''
| Stop when value of the expression ''expr'' changes.
|-
| continue
| Continue execution until a breakpoint or a watchpoint appears.
|-
| backtrace
| Print a list of functions that are currently active.
|-
| quit
| Exit GDB.
|}

'''Example:'''
We debug the following program called bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>

int global = 0;

void begin() {
global = 1;
}

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}

void end() {
global = 2;
}

int main() {
begin();
loop();
end();

return 0;
}
</source>

'''Sample GDB session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27 loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
(gdb) next
27 loop();
(gdb) step
loop () at bug.c:13
13 for(i = 0; i < 8; i++)
(gdb) next
15 k = i/2*2;
(gdb) next
16 v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16 v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit
</pre>

== Core dumps ==
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
<pre>$ ulimit -c unlimited</pre>
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
<pre>$ gdb ./ex core.xxx</pre>
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.

== Multithreaded debugging ==
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step.
The following commands are useful for multithreaded debugging:

{| {{Table}}
|-
! Command
! Description
|-
| info threads
| Shows the status of all existing threads.
|-
| thread ''num''
| Switches to the thread with the number ''num''
|}

'''Example:'''
We debug the following program called thread_bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>
#include <pthread.h>

pthread_t thread;

void* thread3 (void* d)
{
int w[2];
int c, l;

for(c = 0; c < 8; c++) {
l = c/2*2; /* should have been l = c/(2*2); */
w[l] = c;
}

return NULL;
}

void* thread2 (void* d)
{
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
sleep(4);
k = i/(2*2); /* should have been k = i/(2*2); */
v[k] = i;
}

return NULL;
}

int main (){

pthread_create (&thread, NULL, thread2, NULL);
pthread_create (&thread, NULL, thread3, NULL);

//Thread 1
int count1 = 0;

while(count1 < 4000){
printf("Thread 1: %d\n", count1++);
}

pthread_join(thread, NULL);
return 0;
}
</source>

'''Sample GDB thread session:'''
<pre>
$ gcc -g thread_bug.c -o thread_bug -lpthread
$ gdb ./thread_bug
[...]
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break thread3
(gdb) break thread2
(gdb) break main
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/thread_bug

Breakpoint 3, main () at thread_bug.c:35
35 pthread_create (&thread, NULL, thread2, NULL);
(gdb) info threads
* 1 Thread 0x7ffff7fe5700 (LWP 49049) main () at thread_bug.c:35
(gdb) next
[New Thread 0x7ffff7fe3700 (LWP 49077)]
36 pthread_create (&thread, NULL, thread3, NULL);
(gdb) info threads
2 Thread 0x7ffff7fe3700 (LWP 49077) thread2 (d=0x0) at thread_bug.c:24
* 1 Thread 0x7ffff7fe5700 (LWP 49049) main () at thread_bug.c:36
(gdb) next
[Switching to Thread 0x7ffff7fe3700 (LWP 49077)]

Breakpoint 2, thread2 (d=0x0) at thread_bug.c:24
24 for(i = 0; i < 8; i++) {
(gdb) info threads
* 2 Thread 0x7ffff7fe3700 (LWP 49077) thread2 (d=0x0) at thread_bug.c:24
1 Thread 0x7ffff7fe5700 (LWP 49049) 0x000000362f8e55a7 in mprotect () from /lib64/libc.so.6
(gdb) next
25 sleep(4);
(gdb) next
[New Thread 0x7ffff77e2700 (LWP 49086)]
[Switching to Thread 0x7ffff77e2700 (LWP 49086)]

Breakpoint 1, thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) info threads
* 3 Thread 0x7ffff77e2700 (LWP 49086) thread3 (d=0x0) at thread_bug.c:11
2 Thread 0x7ffff7fe3700 (LWP 49077) 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 49049) 0x000000362f874720 in _IO_setb_internal () from /lib64/libc.so.6
(gdb) next
12 l = c/(2*2); /* should have been l = c/(2*2); */
(gdb) next
13 w[l] = c;
(gdb) next
11 for(c = 0; c < 8; c++) {
</pre>

[[Category:debugger software]]

Development/GDB

2014-05-20T14:40:51Z

R Doros: First part of the gdb session

{|{{Softwarebox}}
|-
! colspan="2" style="text-align:center" | GDB: The GNU Project Debugger
|-
| module load
| devel/gdb

|-
| License
| [http://www.gnu.org/licenses/ GPL]
|-
|Citing
|
|-
| Links
| [http://www.gnu.org/software/gdb/ Homepage]; [http://www.gnu.org/software/gdb/documentation/ Documentation]; [https://sourceware.org/gdb/wiki/ Wiki]; [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists]
|
|-
| Graphical Interface
| No 
|-
| Included in module
| 
|}

The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.

== Loading ==
It is possible to choose between several versions of GDB and GCC. By default both programms are installed. You can check the version which is currently used with:
<pre>$ gdb --version</pre>
<pre>$ gcc --version</pre>

To use a different version of GCC or GDB, you have to load it through the module system. For example you need GDB 7.7 or higher for GCC version 4.8 and above. In this case load the devel/gdb module.

How to check which versions are available:
<pre>$ module avail</pre>

How to load the desired version (e.g. GDB version 7.7):
<pre>$ module load devel/gdb/7.7</pre>

If you want to use IDB load the Intel compiler module:
<pre>$ module load compiler/intel</pre>

== Documentation ==
For '''online documentation''' see the links section in the summary table at the top of this page. For '''local documentation''' consult the man page.
<pre>$ man gdb</pre>
or
<pre>$ man idb</pre>

== Basic commands ==
The code you want to debug should be compiled with the -g option. If the optimization flag is not set, the GCC will still do some basic optimization. Therefore, it is recommended to turn off the optmization explicitly with the -O0 parameter for debugging. To start a debug session for a program execute GDB with the program path as parameter:
<pre>$ gdb ./example</pre>

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

{| {{Table}}
|-
! Command
! Description
|-
| help ''cmd''
| Show help for command ''cmd''.
|-
| break ''func''
| Set a breakpoint at function ''func''.
|-
| run
| Start program.
|-
| next
| Go to next program line. Do not enter functions.
|-
| step
| Go to next program line. Enter functions.
|-
| list
| Show the surrounding source code of the currently processed line.
|-
| print ''expr''
| Print the value of the expression ''expr''.
|-
| display ''expr''
| Display the value of the expression ''expr'' every time the program stops.
|-
| watch ''expr''
| Stop when value of the expression ''expr'' changes.
|-
| continue
| Continue execution until a breakpoint or a watchpoint appears.
|-
| backtrace
| Print a list of functions that are currently active.
|-
| quit
| Exit GDB.
|}

'''Example:'''
We debug the following program called bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>

int global = 0;

void begin() {
global = 1;
}

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}

void end() {
global = 2;
}

int main() {
begin();
loop();
end();

return 0;
}
</source>

'''Sample GDB session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27 loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
(gdb) next
27 loop();
(gdb) step
loop () at bug.c:13
13 for(i = 0; i < 8; i++)
(gdb) next
15 k = i/2*2;
(gdb) next
16 v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16 v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit
</pre>

== Core dumps ==
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
<pre>$ ulimit -c unlimited</pre>
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
<pre>$ gdb ./ex core.xxx</pre>
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.

== Multithreaded debugging ==
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step.
The following commands are useful for multithreaded debugging:

{| {{Table}}
|-
! Command
! Description
|-
| info threads
| Shows the status of all existing threads.
|-
| thread ''num''
| Switches to the thread with the number ''num''
|}

'''Example:'''
We debug the following program called thread_bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>
#include <pthread.h>

pthread_t thread;

void* thread3 (void* d)
{
int w[2];
int c, l;

for(c = 0; c < 8; c++) {
l = c/(2*2); /* should have been l = c/(2*2); */
w[l] = c;
}

return NULL;
}

void* thread2 (void* d)
{
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
sleep(4);
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}

return NULL;
}

int main (){

pthread_create (&thread, NULL, thread2, NULL);
pthread_create (&thread, NULL, thread3, NULL);

//Thread 1
int count1 = 0;

while(count1 < 4000){
printf("Thread 1: %d\n", count1++);
}

pthread_join(thread, NULL);
return 0;
}
</source>

'''Sample GDB thread session:'''
<pre>
$ gcc -g thread_bug.c -o thread_bug -lpthread
$ gdb ./thread_bug
[...]
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break thread3
(gdb) break thread2
(gdb) break main
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/thread_bug

Breakpoint 3, main () at thread_bug.c:35
35 pthread_create (&thread, NULL, thread2, NULL);
(gdb) info threads
* 1 Thread 0x7ffff7fe5700 (LWP 49049) main () at thread_bug.c:35
(gdb) next
[New Thread 0x7ffff7fe3700 (LWP 49077)]
36 pthread_create (&thread, NULL, thread3, NULL);
(gdb) info threads
2 Thread 0x7ffff7fe3700 (LWP 49077) thread2 (d=0x0) at thread_bug.c:24
* 1 Thread 0x7ffff7fe5700 (LWP 49049) main () at thread_bug.c:36
(gdb) next
[Switching to Thread 0x7ffff7fe3700 (LWP 49077)]

Breakpoint 2, thread2 (d=0x0) at thread_bug.c:24
24 for(i = 0; i < 8; i++) {
(gdb) info threads
* 2 Thread 0x7ffff7fe3700 (LWP 49077) thread2 (d=0x0) at thread_bug.c:24
1 Thread 0x7ffff7fe5700 (LWP 49049) 0x000000362f8e55a7 in mprotect () from /lib64/libc.so.6
(gdb) next
25 sleep(4);
(gdb) next
[New Thread 0x7ffff77e2700 (LWP 49086)]
[Switching to Thread 0x7ffff77e2700 (LWP 49086)]

Breakpoint 1, thread3 (d=0x0) at thread_bug.c:11
11 for(c = 0; c < 8; c++) {
(gdb) info threads
* 3 Thread 0x7ffff77e2700 (LWP 49086) thread3 (d=0x0) at thread_bug.c:11
2 Thread 0x7ffff7fe3700 (LWP 49077) 0x000000362f8accdd in nanosleep () from /lib64/libc.so.6
1 Thread 0x7ffff7fe5700 (LWP 49049) 0x000000362f874720 in _IO_setb_internal () from /lib64/libc.so.6
(gdb) next
12 l = c/(2*2); /* should have been l = c/(2*2); */
(gdb) next
13 w[l] = c;
(gdb) next
11 for(c = 0; c < 8; c++) {
</pre>

[[Category:debugger software]]

Development/GDB

2014-05-20T14:30:59Z

R Doros: One line source code

{|{{Softwarebox}}
|-
! colspan="2" style="text-align:center" | GDB: The GNU Project Debugger
|-
| module load
| devel/gdb

|-
| License
| [http://www.gnu.org/licenses/ GPL]
|-
|Citing
|
|-
| Links
| [http://www.gnu.org/software/gdb/ Homepage]; [http://www.gnu.org/software/gdb/documentation/ Documentation]; [https://sourceware.org/gdb/wiki/ Wiki]; [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists]
|
|-
| Graphical Interface
| No 
|-
| Included in module
| 
|}

The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.

== Loading ==
It is possible to choose between several versions of GDB and GCC. By default both programms are installed. You can check the version which is currently used with:
<pre>$ gdb --version</pre>
<pre>$ gcc --version</pre>

To use a different version of GCC or GDB, you have to load it through the module system. For example you need GDB 7.7 or higher for GCC version 4.8 and above. In this case load the devel/gdb module.

How to check which versions are available:
<pre>$ module avail</pre>

How to load the desired version (e.g. GDB version 7.7):
<pre>$ module load devel/gdb/7.7</pre>

If you want to use IDB load the Intel compiler module:
<pre>$ module load compiler/intel</pre>

== Documentation ==
For '''online documentation''' see the links section in the summary table at the top of this page. For '''local documentation''' consult the man page.
<pre>$ man gdb</pre>
or
<pre>$ man idb</pre>

== Basic commands ==
The code you want to debug should be compiled with the -g option. If the optimization flag is not set, the GCC will still do some basic optimization. Therefore, it is recommended to turn off the optmization explicitly with the -O0 parameter for debugging. To start a debug session for a program execute GDB with the program path as parameter:
<pre>$ gdb ./example</pre>

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

{| {{Table}}
|-
! Command
! Description
|-
| help ''cmd''
| Show help for command ''cmd''.
|-
| break ''func''
| Set a breakpoint at function ''func''.
|-
| run
| Start program.
|-
| next
| Go to next program line. Do not enter functions.
|-
| step
| Go to next program line. Enter functions.
|-
| list
| Show the surrounding source code of the currently processed line.
|-
| print ''expr''
| Print the value of the expression ''expr''.
|-
| display ''expr''
| Display the value of the expression ''expr'' every time the program stops.
|-
| watch ''expr''
| Stop when value of the expression ''expr'' changes.
|-
| continue
| Continue execution until a breakpoint or a watchpoint appears.
|-
| backtrace
| Print a list of functions that are currently active.
|-
| quit
| Exit GDB.
|}

'''Example:'''
We debug the following program called bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>

int global = 0;

void begin() {
global = 1;
}

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}

void end() {
global = 2;
}

int main() {
begin();
loop();
end();

return 0;
}
</source>

'''Sample GDB session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27 loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
(gdb) next
27 loop();
(gdb) step
loop () at bug.c:13
13 for(i = 0; i < 8; i++)
(gdb) next
15 k = i/2*2;
(gdb) next
16 v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16 v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit
</pre>

== Core dumps ==
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
<pre>$ ulimit -c unlimited</pre>
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
<pre>$ gdb ./ex core.xxx</pre>
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.

== Multithreaded debugging ==
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step.
The following commands are useful for multithreaded debugging:

{| {{Table}}
|-
! Command
! Description
|-
| info threads
| Shows the status of all existing threads.
|-
| thread ''num''
| Switches to the thread with the number ''num''
|}

'''Example:'''
We debug the following program called thread_bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>
#include <pthread.h>

pthread_t thread;

void* thread3 (void* d)
{
int w[2];
int c, l;

for(c = 0; c < 8; c++) {
l = c/(2*2); /* should have been l = c/(2*2); */
w[l] = c;
}

return NULL;
}

void* thread2 (void* d)
{
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
sleep(4);
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}

return NULL;
}

int main (){

pthread_create (&thread, NULL, thread2, NULL);
pthread_create (&thread, NULL, thread3, NULL);

//Thread 1
int count1 = 0;

while(count1 < 4000){
printf("Thread 1: %d\n", count1++);
}

pthread_join(thread, NULL);
return 0;
}
</source>

'''Sample GDB thread session:'''
<pre>
$ gcc -g thread_bug.c -o thread_bug -lpthread
$ gdb ./thread_bug
</pre>

[[Category:debugger software]]

Development/GDB

2014-05-20T14:10:31Z

R Doros: Start session

{|{{Softwarebox}}
|-
! colspan="2" style="text-align:center" | GDB: The GNU Project Debugger
|-
| module load
| devel/gdb

|-
| License
| [http://www.gnu.org/licenses/ GPL]
|-
|Citing
|
|-
| Links
| [http://www.gnu.org/software/gdb/ Homepage]; [http://www.gnu.org/software/gdb/documentation/ Documentation]; [https://sourceware.org/gdb/wiki/ Wiki]; [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists]
|
|-
| Graphical Interface
| No 
|-
| Included in module
| 
|}

The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.

== Loading ==
It is possible to choose between several versions of GDB and GCC. By default both programms are installed. You can check the version which is currently used with:
<pre>$ gdb --version</pre>
<pre>$ gcc --version</pre>

To use a different version of GCC or GDB, you have to load it through the module system. For example you need GDB 7.7 or higher for GCC version 4.8 and above. In this case load the devel/gdb module.

How to check which versions are available:
<pre>$ module avail</pre>

How to load the desired version (e.g. GDB version 7.7):
<pre>$ module load devel/gdb/7.7</pre>

If you want to use IDB load the Intel compiler module:
<pre>$ module load compiler/intel</pre>

== Documentation ==
For '''online documentation''' see the links section in the summary table at the top of this page. For '''local documentation''' consult the man page.
<pre>$ man gdb</pre>
or
<pre>$ man idb</pre>

== Basic commands ==
The code you want to debug should be compiled with the -g option. If the optimization flag is not set, the GCC will still do some basic optimization. Therefore, it is recommended to turn off the optmization explicitly with the -O0 parameter for debugging. To start a debug session for a program execute GDB with the program path as parameter:
<pre>$ gdb ./example</pre>

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

{| {{Table}}
|-
! Command
! Description
|-
| help ''cmd''
| Show help for command ''cmd''.
|-
| break ''func''
| Set a breakpoint at function ''func''.
|-
| run
| Start program.
|-
| next
| Go to next program line. Do not enter functions.
|-
| step
| Go to next program line. Enter functions.
|-
| list
| Show the surrounding source code of the currently processed line.
|-
| print ''expr''
| Print the value of the expression ''expr''.
|-
| display ''expr''
| Display the value of the expression ''expr'' every time the program stops.
|-
| watch ''expr''
| Stop when value of the expression ''expr'' changes.
|-
| continue
| Continue execution until a breakpoint or a watchpoint appears.
|-
| backtrace
| Print a list of functions that are currently active.
|-
| quit
| Exit GDB.
|}

'''Example:'''
We debug the following program called bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>

int global = 0;

void begin() {
global = 1;
}

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}

void end() {
global = 2;
}

int main() {
begin();
loop();
end();

return 0;
}
</source>

'''Sample GDB session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27 loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
(gdb) next
27 loop();
(gdb) step
loop () at bug.c:13
13 for(i = 0; i < 8; i++)
(gdb) next
15 k = i/2*2;
(gdb) next
16 v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16 v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit
</pre>

== Core dumps ==
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
<pre>$ ulimit -c unlimited</pre>
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
<pre>$ gdb ./ex core.xxx</pre>
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.

== Multithreaded debugging ==
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step.
The following commands are useful for multithreaded debugging:

{| {{Table}}
|-
! Command
! Description
|-
| info threads
| Shows the status of all existing threads.
|-
| thread ''num''
| Switches to the thread with the number ''num''
|}

'''Example:'''
We debug the following program called thread_bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>
#include <pthread.h>

pthread_t thread;

void* thread3 (void* d)
{
int w[2];
int c, l;

for(c = 0; c < 8; c++) {
l = c/(2*2); /* should have been l = c/(2*2); */
w[l] = c;
}

return NULL;
}

void* thread2 (void* d)
{
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}

return NULL;
}

int main (){

pthread_create (&thread, NULL, thread2, NULL);
pthread_create (&thread, NULL, thread3, NULL);

//Thread 1
int count1 = 0;

while(count1 < 4000){
printf("Thread 1: %d\n", count1++);
}

pthread_join(thread, NULL);
return 0;
}
</source>

'''Sample GDB thread session:'''
<pre>
$ gcc -g thread_bug.c -o thread_bug -lpthread
$ gdb ./thread_bug
</pre>

[[Category:debugger software]]

Development/GDB

2014-05-20T14:06:40Z

R Doros: Include thread example

{|{{Softwarebox}}
|-
! colspan="2" style="text-align:center" | GDB: The GNU Project Debugger
|-
| module load
| devel/gdb

|-
| License
| [http://www.gnu.org/licenses/ GPL]
|-
|Citing
|
|-
| Links
| [http://www.gnu.org/software/gdb/ Homepage]; [http://www.gnu.org/software/gdb/documentation/ Documentation]; [https://sourceware.org/gdb/wiki/ Wiki]; [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists]
|
|-
| Graphical Interface
| No 
|-
| Included in module
| 
|}

The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.

== Loading ==
It is possible to choose between several versions of GDB and GCC. By default both programms are installed. You can check the version which is currently used with:
<pre>$ gdb --version</pre>
<pre>$ gcc --version</pre>

To use a different version of GCC or GDB, you have to load it through the module system. For example you need GDB 7.7 or higher for GCC version 4.8 and above. In this case load the devel/gdb module.

How to check which versions are available:
<pre>$ module avail</pre>

How to load the desired version (e.g. GDB version 7.7):
<pre>$ module load devel/gdb/7.7</pre>

If you want to use IDB load the Intel compiler module:
<pre>$ module load compiler/intel</pre>

== Documentation ==
For '''online documentation''' see the links section in the summary table at the top of this page. For '''local documentation''' consult the man page.
<pre>$ man gdb</pre>
or
<pre>$ man idb</pre>

== Basic commands ==
The code you want to debug should be compiled with the -g option. If the optimization flag is not set, the GCC will still do some basic optimization. Therefore, it is recommended to turn off the optmization explicitly with the -O0 parameter for debugging. To start a debug session for a program execute GDB with the program path as parameter:
<pre>$ gdb ./example</pre>

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

{| {{Table}}
|-
! Command
! Description
|-
| help ''cmd''
| Show help for command ''cmd''.
|-
| break ''func''
| Set a breakpoint at function ''func''.
|-
| run
| Start program.
|-
| next
| Go to next program line. Do not enter functions.
|-
| step
| Go to next program line. Enter functions.
|-
| list
| Show the surrounding source code of the currently processed line.
|-
| print ''expr''
| Print the value of the expression ''expr''.
|-
| display ''expr''
| Display the value of the expression ''expr'' every time the program stops.
|-
| watch ''expr''
| Stop when value of the expression ''expr'' changes.
|-
| continue
| Continue execution until a breakpoint or a watchpoint appears.
|-
| backtrace
| Print a list of functions that are currently active.
|-
| quit
| Exit GDB.
|}

'''Example:'''
We debug the following program called bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>

int global = 0;

void begin() {
global = 1;
}

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}

void end() {
global = 2;
}

int main() {
begin();
loop();
end();

return 0;
}
</source>

'''Sample GDB session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27 loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
(gdb) next
27 loop();
(gdb) step
loop () at bug.c:13
13 for(i = 0; i < 8; i++)
(gdb) next
15 k = i/2*2;
(gdb) next
16 v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16 v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit
</pre>

== Core dumps ==
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
<pre>$ ulimit -c unlimited</pre>
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
<pre>$ gdb ./ex core.xxx</pre>
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.

== Multithreaded debugging ==
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step.
The following commands are useful for multithreaded debugging:

{| {{Table}}
|-
! Command
! Description
|-
| info threads
| Shows the status of all existing threads.
|-
| thread ''num''
| Switches to the thread with the number ''num''
|}

'''Example:'''
We debug the following program called thread_bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>
#include <pthread.h>

pthread_t thread;

void* thread3 (void* d)
{
int w[2];
int c, l;

for(c = 0; c < 8; c++) {
l = c/(2*2); /* should have been l = c/(2*2); */
w[l] = c;
}

return NULL;
}

void* thread2 (void* d)
{
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}

return NULL;
}

int main (){

pthread_create (&thread, NULL, thread2, NULL);
pthread_create (&thread, NULL, thread3, NULL);

//Thread 1
int count1 = 0;

while(count1 < 4000){
printf("Thread 1: %d\n", count1++);
}

pthread_join(thread, NULL);
return 0;
}
</source>

[[Category:debugger software]]

Development/GDB

2014-03-20T08:37:21Z

R Doros: Versionnumbers removed, more infos for gdb debugging insert

{| style="border-style: solid; border-width: 1px"
! Navigation: [[BwHPC_Best_Practices_Repository|bwHPC BPR]]
|}


{|{{Softwarebox}}
|-
! colspan="2" style="text-align:center" | GDB: The GNU Project Debugger
|-
| module load
| devel/gdb

|-
| License
| [http://www.gnu.org/licenses/ GPL]
|-
|Citing
|
|-
| Links
| [http://www.gnu.org/software/gdb/ Homepage]; [http://www.gnu.org/software/gdb/documentation/ Documentation]; [https://sourceware.org/gdb/wiki/ Wiki]; [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists]
|
|-
| Graphical Interface
| No 
|-
| Included in module
| 
|}

= Debugging =

The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.

== Loading ==
It is possible to choose between several versions of GDB and GCC. By default both programms are installed. You can check the version which is currently used with:
<pre>$ gdb --version</pre>
<pre>$ gcc --version</pre>

If you want to use another version of GCC or GDB, you have to load it through the module system. For example you need GDB 7.7 or higher for GCC version 4.8 and above. In this case load the devel/gdb module.

How to check which versions are available:
<pre>$ module avail</pre>

How to load the desired version (e.g. GDB version 7.7):
<pre>$ module load devel/gdb/7.7</pre>

If you want to use IDB load the Intel compiler module:
<pre>$ module load compiler/intel</pre>

== Documentation ==
For '''online documentation''' see the links section in the summary table at the top of this page. For '''local documentation''' consult the man page.
<pre>$ man gdb</pre>
or
<pre>$ man idb</pre>

== Basic commands ==
The code you want to debug should be compiled with the -g option. If the optimization flag is just not set, the GCC will still do some basic optimization. So it is recommended to turn off the optmization explicitly with the -O0 parameter. To start a debug session for a program execute GDB with the program path as parameter:
<pre>$ gdb ./example</pre>

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

{| {{Table}}
|-
! Command
! Description
|-
| help ''cmd''
| Show help for command ''cmd''.
|-
| break ''func''
| Set a breakpoint at function ''func''.
|-
| run
| Start program.
|-
| next
| Go to next program line. Do not enter functions.
|-
| step
| Go to next program line. Enter functions.
|-
| list
| Show the surrounding source code of the currently processed line.
|-
| print ''expr''
| Print the value of the expression ''expr''.
|-
| display ''expr''
| Display the value of the expression ''expr'' every time the program stops.
|-
| watch ''expr''
| Stop when value of the expression ''expr'' changes.
|-
| continue
| Continue execution until a breakpoint or a watchpoint appears.
|-
| backtrace
| Print a list of functions that are currently active.
|-
| quit
| Exit GDB.
|}

'''Example:'''
We debug the following program called bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>

int global = 0;

void begin() {
global = 1;
}

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}

void end() {
global = 2;
}

int main() {
begin();
loop();
end();

return 0;
}
</source>

'''Sample GDB session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27 loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
(gdb) next
27 loop();
(gdb) step
loop () at bug.c:13
13 for(i = 0; i < 8; i++)
(gdb) next
15 k = i/2*2;
(gdb) next
16 v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16 v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit
</pre>

== Core dumps ==
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
<pre>$ ulimit -c unlimited</pre>
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
<pre>$ gdb ./ex core.xxx</pre>
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.

== Multithreaded debugging ==
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step.
The following commands are useful for multithreaded debugging:

{| {{Table}}
|-
! Command
! Description
|-
| info threads
| Shows the status of all existing threads.
|-
| thread ''num''
| Switches to the thread with the number ''num''
|}

= Valgrind =

Valgrind can be used for debugging and profiling programs. It ships with various tools, some of them are described in the following table.

{| {{Table}}
|-
! Tool
! Description
|-
| memcheck
| Detects heap array overruns, memory leaks and incorrect freeing of heap memory.
|-
| exp-sgcheck
| Detects stack and global array overruns.
|-
| callgrind
| Profiler. Helps finding bottlenecks.
|-
| helgrind
| Detects race conditions and deadlocks.
|-
| drd
| Detects race conditions. Does not detect deadlocks but needs less memory than helgrind.
|}

'''Loading:''' There is no need for loading valgrind because it is available by default.

'''Online documentation:''' http://valgrind.org/docs/
'''Local documentation:''' <pre>man valgrind</pre>

'''Usage:'''
To simulate your program and gather information about it using a specific tool, valgrind can be called like this
<pre>$ valgrind --tool=<tool> --log-file=<logfile> ./example</pre>
For example to check for race conditions the command would look like this:
<pre>$ valgrind --tool=drd --log-file=log.mem ./example</pre>

Beware that the simulation via valgrind can take much longer and can consume much more memory compared to a normal execution.
Furthermore the tools can give false positives, i.e. they report errors even though the are none.

'''Example for helgrind and drd:'''

This small program demonstrates how to detect data races in parallel programs. Both helgrind and drd support POSIX Threads (Pthreads) but not OpenMP so prepare to get many false positives when using these tools with OpenMP.

<source lang=c>
/* pthreads.c */
#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
#include <math.h>

#define NUM_THREADS 2

int v = 0;

void* f(void* x)
{
int j = 0;
int k = *(int*)x;

for(j = 0; j < 100; j++)
v += sin(j) + 1 + k;

return NULL;
}

int main(int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int idx[NUM_THREADS];
int t;

for(t=0; t<NUM_THREADS; t++) {
idx[t] = t;
pthread_create(&threads[t], NULL, f, (void*)&idx[t]);
}

for(t=0; t<NUM_THREADS; t++)
pthread_join(threads[t], NULL);

printf("%i\n", v);

pthread_exit(NULL);

return 0;
}
</source>
The program can be compiled with
<pre>$ gcc -Wall pthreads.c -o pthreads -pthread -lm -O2</pre>
where -pthread tells the compiler to link against the Pthreads library and -lm is needed for linking against the math library.
This program has a race condition, consecutive executions can give different outputs. Such errors are hard to find because they can show up very rarely.
Helgrind can tell you the problematic part of your code:
<pre>
$ valgrind --tool=helgrind ./pthreads
...
==29237== ----------------------------------------------------------------
==29237==
==29237== Possible data race during read of size 4 at 0x600C78 by thread #3
==29237== Locks held: none
==29237== at 0x4007BC: f(void*) (in /pfs/data2/home/xx/xxxx/xxxx/pthreads)
==29237== by 0x4A0C0D4: mythread_wrapper (hg_intercepts.c:219)
==29237== by 0x3EDA6079D0: start_thread (in /lib64/libpthread-2.12.so)
==29237== by 0x3ED9AE8B6C: clone (in /lib64/libc-2.12.so)
==29237==
==29237== This conflicts with a previous write of size 4 by thread #2
==29237== Locks held: none
==29237== at 0x4007FB: f(void*) (in /pfs/data2/home/xx/xxxx/xxxx/pthreads)
==29237== by 0x4A0C0D4: mythread_wrapper (hg_intercepts.c:219)
==29237== by 0x3EDA6079D0: start_thread (in /lib64/libpthread-2.12.so)
==29237== by 0x3ED9AE8B6C: clone (in /lib64/libc-2.12.so)
==29237==
==29237== ----------------------------------------------------------------
...
</pre>
Now we know that we have to check the function f for a possible data race. A similar output can be achieved using drd with the following command:
<pre>$ valgrind --tool=drd ./pthreads</pre>

Valgrind can give even better hints when compiled with debug information (helgrind produces similar output):
<pre>
$ gcc -g -Wall pthreads.c -o pthreads -pthread -lm
$ valgrind --tool=drd ./pthreads
...
==604== Thread 3:
==604== Conflicting load by thread 3 at 0x00600cc0 size 4
==604== at 0x400749: f(void*) (pthreads.c:17)
==604== by 0x4A128B4: vgDrd_thread_wrapper (drd_pthread_intercepts.c:355)
==604== by 0x3EDA6079D0: start_thread (in /lib64/libpthread-2.12.so)
==604== by 0x3ED9AE8B6C: clone (in /lib64/libc-2.12.so)
==604== Allocation context: BSS section of fs/data2/home/xx/xxxx/xxxx/pthreads
==604== Other segment start (thread 2)
==604== (thread finished, call stack no longer available)
==604== Other segment end (thread 2)
==604== (thread finished, call stack no longer available)
...
</pre>
The execution can take longer but now we know that the problem is most likely in line 17 (which is correct).

= Parallel Debugger ddt =

{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #e7aa01;"
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #e7aa01; text-align:left; color:#000; padding:0.2em 0.4em;">Currently only for employees of KIT</div>
|-
| style="color:#000;" | <div style="padding:2px 5px">On bwUniCluster the GUI based distributed debugging tool (ddt) may be used to debug serial as
well as parallel applications. For serial applications also the GNU gdb or Intel idb debugger
may be used. The Intel idb comes with the compiler and information on this tool is available
together with the compiler documentation. In order to debug your program it must be
compiled and linked using the -g compiler option. This will force the compiler to add additional information to the object code which is used by the debugger at runtime.
</div>
|}

ddt consists of a graphical frontend and a backend serial debugger which controls the
application program. One instance of the serial debugger controls one MPI process. Via the
frontend the user interacts with the debugger to select the program that will be debugged,
to specify different options and to monitor the execution of the program. Debugging
commands may be sent to one, all or a subset of the MPI processes.

Before the parallel debugger ddt can be used, it is necessary to load the corresponding
module file:
<pre>
$ module use /opt/bwhpc/ka/modulefiles (only available for employees of KIT)
$ module add debugger/ddt
</pre>

Now ddt may be started with the command
<pre>
$ ddt program
</pre>

where program is the name of your program that you want to debug.

[[File:ddt1_750.jpg]]

Figure: DDT startup window

The above figure shows ddt’s startup window. Before actually starting the debugging session
you should check the contents of several fields in this window:

1. The top line shows the executable file that will be run under control of the debugger. In
the following lines you may input some options that are passed to your program or to the
MPI environment.

2. If your program reads data from stdin you can specify an input file in the startup window.

3. Before starting an MPI program you should check that "Open MPI (Compatability)" or
"Intel MPI" is the MPI implementation that has been selected. If this is not the case, you
have to change this. Otherwise ddt may not be able to run your program. In order to debug
serial programs, the selected MPI implementation should be "none". You may also change
the underlying serial debugger using the "change" button. By default ddt uses its own serial debugger, but it may also use the Intel idb debugger.

4. Select the number of MPI processes that will be started by ddt. If you are using ddt within
a batch job, replace mpirun by ddt in the command line of ????? and make sure that the
chosen number of MPI processes is identical to the number of MPI tasks (-p option ???) that
you selected with the ?????? command. When you debug a serial program, select 1.

5. After you have checked all inputs in the ddt startup window, you can start the debugging
session by pressing the "run" button.

The ddt window now shows the source code of the program that is being debugged and breakpoints can be set by just pointing to the corresponding line and pressing the right
mouse button. So you may step through your program, display the values of variables
and arrays and look at the message queues.

[[File:ddt2_750.jpg]]

Development/GDB

2014-03-04T18:21:29Z

R Doros: Section documentation update

{| style="border-style: solid; border-width: 1px"
! Navigation: [[BwHPC_Best_Practices_Repository|bwHPC BPR]]
|}


{|{{Softwarebox}}
|-
! colspan="2" style="text-align:center" | GDB: The GNU Project Debugger
|-
| module load
| devel/gdb

|-
| License
| [http://www.gnu.org/licenses/ GPL]
|-
|Citing
|
|-
| Links
| [http://www.gnu.org/software/gdb/ Homepage]; [http://www.gnu.org/software/gdb/documentation/ Documentation]; [https://sourceware.org/gdb/wiki/ Wiki]; [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists]
|
|-
| Graphical Interface
| No 
|-
| Included in module
| 
|}

= Debugging =

The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.

== Loading ==
By default GCC 4.4 and GDB 7.2 are available. If you want to use
another version of GCC, you have to load it through the module system. For GCC
version 4.8 and above
you need GDB 7.7 or higher. In this case load the devel/gdb module:
<pre>$ module load devel/gdb</pre>

You can check the version which is currently used with:
<pre>$ gdb --version</pre>

If you want to use IDB load the Intel compiler module:
<pre>$ module load compiler/intel</pre>

== Documentation ==
For '''online documentation''' see the links section in the summary table at the top of this page. For '''local documentation''' consult the man page.
<pre>$ man gdb</pre>
or
<pre>$ man idb</pre>

== Basic commands ==
The code you want to debug should be compiled with the -g option and it is recommended that optimization flags are not set. To start a debug session for a program execute GDB with the program path as parameter:
<pre>$ gdb ./example</pre>

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

{| {{Table}}
|-
! Command
! Description
|-
| help ''cmd''
| Show help for command ''cmd''.
|-
| break ''func''
| Set a breakpoint at function ''func''.
|-
| run
| Start program.
|-
| next
| Go to next program line. Do not enter functions.
|-
| step
| Go to next program line. Enter functions.
|-
| list
| Show the surrounding source code of the currently processed line.
|-
| print ''expr''
| Print the value of the expression ''expr''.
|-
| display ''expr''
| Display the value of the expression ''expr'' every time the program stops.
|-
| watch ''expr''
| Stop when value of the expression ''expr'' changes.
|-
| continue
| Continue execution until a breakpoint or a watchpoint appears.
|-
| backtrace
| Print a list of functions that are currently active.
|-
| quit
| Exit GDB.
|}

'''Example:'''
We debug the following program called bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>

int global = 0;

void begin() {
global = 1;
}

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}

void end() {
global = 2;
}

int main() {
begin();
loop();
end();

return 0;
}
</source>

'''Sample GDB session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27 loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
(gdb) next
27 loop();
(gdb) step
loop () at bug.c:13
13 for(i = 0; i < 8; i++)
(gdb) next
15 k = i/2*2;
(gdb) next
16 v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16 v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit
</pre>

== Core dumps ==
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
<pre>$ ulimit -c unlimited</pre>
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
<pre>$ gdb ./ex core.xxx</pre>
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.

== Multithreaded debugging ==
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step.
The following commands are useful for multithreaded debugging:

{| {{Table}}
|-
! Command
! Description
|-
| info threads
| Shows the status of all existing threads.
|-
| thread ''num''
| Switches to the thread with the number ''num''
|}

= Valgrind =

Valgrind can be used for debugging and profiling programs. It ships with various tools, some of them are described in the following table.

{| {{Table}}
|-
! Tool
! Description
|-
| memcheck
| Detects heap array overruns, memory leaks and incorrect freeing of heap memory.
|-
| exp-sgcheck
| Detects stack and global array overruns.
|-
| callgrind
| Profiler. Helps finding bottlenecks.
|-
| helgrind
| Detects race conditions and deadlocks.
|-
| drd
| Detects race conditions. Does not detect deadlocks but needs less memory than helgrind.
|}

'''Loading:''' There is no need for loading valgrind because it is available by default.

'''Online documentation:''' http://valgrind.org/docs/
'''Local documentation:''' <pre>man valgrind</pre>

'''Usage:'''
To simulate your program and gather information about it using a specific tool, valgrind can be called like this
<pre>$ valgrind --tool=<tool> --log-file=<logfile> ./example</pre>
For example to check for race conditions the command would look like this:
<pre>$ valgrind --tool=drd --log-file=log.mem ./example</pre>

Beware that the simulation via valgrind can take much longer and can consume much more memory compared to a normal execution.
Furthermore the tools can give false positives, i.e. they report errors even though the are none.

'''Example for helgrind and drd:'''

This small program demonstrates how to detect data races in parallel programs. Both helgrind and drd support POSIX Threads (Pthreads) but not OpenMP so prepare to get many false positives when using these tools with OpenMP.

<source lang=c>
/* pthreads.c */
#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
#include <math.h>

#define NUM_THREADS 2

int v = 0;

void* f(void* x)
{
int j = 0;
int k = *(int*)x;

for(j = 0; j < 100; j++)
v += sin(j) + 1 + k;

return NULL;
}

int main(int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int idx[NUM_THREADS];
int t;

for(t=0; t<NUM_THREADS; t++) {
idx[t] = t;
pthread_create(&threads[t], NULL, f, (void*)&idx[t]);
}

for(t=0; t<NUM_THREADS; t++)
pthread_join(threads[t], NULL);

printf("%i\n", v);

pthread_exit(NULL);

return 0;
}
</source>
The program can be compiled with
<pre>$ gcc -Wall pthreads.c -o pthreads -pthread -lm -O2</pre>
where -pthread tells the compiler to link against the Pthreads library and -lm is needed for linking against the math library.
This program has a race condition, consecutive executions can give different outputs. Such errors are hard to find because they can show up very rarely.
Helgrind can tell you the problematic part of your code:
<pre>
$ valgrind --tool=helgrind ./pthreads
...
==29237== ----------------------------------------------------------------
==29237==
==29237== Possible data race during read of size 4 at 0x600C78 by thread #3
==29237== Locks held: none
==29237== at 0x4007BC: f(void*) (in /pfs/data2/home/xx/xxxx/xxxx/pthreads)
==29237== by 0x4A0C0D4: mythread_wrapper (hg_intercepts.c:219)
==29237== by 0x3EDA6079D0: start_thread (in /lib64/libpthread-2.12.so)
==29237== by 0x3ED9AE8B6C: clone (in /lib64/libc-2.12.so)
==29237==
==29237== This conflicts with a previous write of size 4 by thread #2
==29237== Locks held: none
==29237== at 0x4007FB: f(void*) (in /pfs/data2/home/xx/xxxx/xxxx/pthreads)
==29237== by 0x4A0C0D4: mythread_wrapper (hg_intercepts.c:219)
==29237== by 0x3EDA6079D0: start_thread (in /lib64/libpthread-2.12.so)
==29237== by 0x3ED9AE8B6C: clone (in /lib64/libc-2.12.so)
==29237==
==29237== ----------------------------------------------------------------
...
</pre>
Now we know that we have to check the function f for a possible data race. A similar output can be achieved using drd with the following command:
<pre>$ valgrind --tool=drd ./pthreads</pre>

Valgrind can give even better hints when compiled with debug information (helgrind produces similar output):
<pre>
$ gcc -g -Wall pthreads.c -o pthreads -pthread -lm
$ valgrind --tool=drd ./pthreads
...
==604== Thread 3:
==604== Conflicting load by thread 3 at 0x00600cc0 size 4
==604== at 0x400749: f(void*) (pthreads.c:17)
==604== by 0x4A128B4: vgDrd_thread_wrapper (drd_pthread_intercepts.c:355)
==604== by 0x3EDA6079D0: start_thread (in /lib64/libpthread-2.12.so)
==604== by 0x3ED9AE8B6C: clone (in /lib64/libc-2.12.so)
==604== Allocation context: BSS section of fs/data2/home/xx/xxxx/xxxx/pthreads
==604== Other segment start (thread 2)
==604== (thread finished, call stack no longer available)
==604== Other segment end (thread 2)
==604== (thread finished, call stack no longer available)
...
</pre>
The execution can take longer but now we know that the problem is most likely in line 17 (which is correct).

= Parallel Debugger ddt =

{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #e7aa01;"
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #e7aa01; text-align:left; color:#000; padding:0.2em 0.4em;">Currently only for employees of KIT</div>
|-
| style="color:#000;" | <div style="padding:2px 5px">On bwUniCluster the GUI based distributed debugging tool (ddt) may be used to debug serial as
well as parallel applications. For serial applications also the GNU gdb or Intel idb debugger
may be used. The Intel idb comes with the compiler and information on this tool is available
together with the compiler documentation. In order to debug your program it must be
compiled and linked using the -g compiler option. This will force the compiler to add additional information to the object code which is used by the debugger at runtime.
</div>
|}

ddt consists of a graphical frontend and a backend serial debugger which controls the
application program. One instance of the serial debugger controls one MPI process. Via the
frontend the user interacts with the debugger to select the program that will be debugged,
to specify different options and to monitor the execution of the program. Debugging
commands may be sent to one, all or a subset of the MPI processes.

Before the parallel debugger ddt can be used, it is necessary to load the corresponding
module file:
<pre>
$ module use /opt/bwhpc/ka/modulefiles (only available for employees of KIT)
$ module add debugger/ddt
</pre>

Now ddt may be started with the command
<pre>
$ ddt program
</pre>

where program is the name of your program that you want to debug.

[[File:ddt1_750.jpg]]

Figure: DDT startup window

The above figure shows ddt’s startup window. Before actually starting the debugging session
you should check the contents of several fields in this window:

1. The top line shows the executable file that will be run under control of the debugger. In
the following lines you may input some options that are passed to your program or to the
MPI environment.

2. If your program reads data from stdin you can specify an input file in the startup window.

3. Before starting an MPI program you should check that "Open MPI (Compatability)" or
"Intel MPI" is the MPI implementation that has been selected. If this is not the case, you
have to change this. Otherwise ddt may not be able to run your program. In order to debug
serial programs, the selected MPI implementation should be "none". You may also change
the underlying serial debugger using the "change" button. By default ddt uses its own serial debugger, but it may also use the Intel idb debugger.

4. Select the number of MPI processes that will be started by ddt. If you are using ddt within
a batch job, replace mpirun by ddt in the command line of ????? and make sure that the
chosen number of MPI processes is identical to the number of MPI tasks (-p option ???) that
you selected with the ?????? command. When you debug a serial program, select 1.

5. After you have checked all inputs in the ddt startup window, you can start the debugging
session by pressing the "run" button.

The ddt window now shows the source code of the program that is being debugged and breakpoints can be set by just pointing to the corresponding line and pressing the right
mouse button. So you may step through your program, display the values of variables
and arrays and look at the message queues.

[[File:ddt2_750.jpg]]

Development/GDB

2014-03-04T18:12:52Z

R Doros: Load section update for gdb 7.7

{| style="border-style: solid; border-width: 1px"
! Navigation: [[BwHPC_Best_Practices_Repository|bwHPC BPR]]
|}


{|{{Softwarebox}}
|-
! colspan="2" style="text-align:center" | GDB: The GNU Project Debugger
|-
| module load
| devel/gdb

|-
| License
| [http://www.gnu.org/licenses/ GPL]
|-
|Citing
|
|-
| Links
| [http://www.gnu.org/software/gdb/ Homepage]; [http://www.gnu.org/software/gdb/documentation/ Documentation]; [https://sourceware.org/gdb/wiki/ Wiki]; [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists]
|
|-
| Graphical Interface
| No 
|-
| Included in module
| 
|}

= Debugging =

The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.

== Loading ==
By default GCC 4.4 and GDB 7.2 are available. If you want to use
another version of GCC, you have to load it through the module system. For GCC
version 4.8 and above
you need GDB 7.7 or higher. In this case load the devel/gdb module:
<pre>$ module load devel/gdb</pre>

You can check the version which is currently used with:
<pre>$ gdb --version</pre>

If you want to use IDB load the Intel compiler module:
<pre>$ module load compiler/intel</pre>

== Documentation ==
'''Online documentation:''' [http://www.gnu.org/software/gdb/documentation/ GDB documentation] and [http://software.intel.com/sites/products/documentation/doclib/stdxe/2013/composerxe/debugger/user_guide/index.htm IDB Documentation]

'''Local documentation:'''
For detailed lists of the different program options consult the man page:
<pre>$ man gdb</pre>
or
<pre>$ man idb</pre>

== Basic commands ==
The code you want to debug should be compiled with the -g option and it is recommended that optimization flags are not set. To start a debug session for a program execute GDB with the program path as parameter:
<pre>$ gdb ./example</pre>

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

{| {{Table}}
|-
! Command
! Description
|-
| help ''cmd''
| Show help for command ''cmd''.
|-
| break ''func''
| Set a breakpoint at function ''func''.
|-
| run
| Start program.
|-
| next
| Go to next program line. Do not enter functions.
|-
| step
| Go to next program line. Enter functions.
|-
| list
| Show the surrounding source code of the currently processed line.
|-
| print ''expr''
| Print the value of the expression ''expr''.
|-
| display ''expr''
| Display the value of the expression ''expr'' every time the program stops.
|-
| watch ''expr''
| Stop when value of the expression ''expr'' changes.
|-
| continue
| Continue execution until a breakpoint or a watchpoint appears.
|-
| backtrace
| Print a list of functions that are currently active.
|-
| quit
| Exit GDB.
|}

'''Example:'''
We debug the following program called bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>

int global = 0;

void begin() {
global = 1;
}

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}

void end() {
global = 2;
}

int main() {
begin();
loop();
end();

return 0;
}
</source>

'''Sample GDB session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27 loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
(gdb) next
27 loop();
(gdb) step
loop () at bug.c:13
13 for(i = 0; i < 8; i++)
(gdb) next
15 k = i/2*2;
(gdb) next
16 v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16 v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit
</pre>

== Core dumps ==
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
<pre>$ ulimit -c unlimited</pre>
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
<pre>$ gdb ./ex core.xxx</pre>
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.

== Multithreaded debugging ==
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step.
The following commands are useful for multithreaded debugging:

{| {{Table}}
|-
! Command
! Description
|-
| info threads
| Shows the status of all existing threads.
|-
| thread ''num''
| Switches to the thread with the number ''num''
|}

= Valgrind =

Valgrind can be used for debugging and profiling programs. It ships with various tools, some of them are described in the following table.

{| {{Table}}
|-
! Tool
! Description
|-
| memcheck
| Detects heap array overruns, memory leaks and incorrect freeing of heap memory.
|-
| exp-sgcheck
| Detects stack and global array overruns.
|-
| callgrind
| Profiler. Helps finding bottlenecks.
|-
| helgrind
| Detects race conditions and deadlocks.
|-
| drd
| Detects race conditions. Does not detect deadlocks but needs less memory than helgrind.
|}

'''Loading:''' There is no need for loading valgrind because it is available by default.

'''Online documentation:''' http://valgrind.org/docs/
'''Local documentation:''' <pre>man valgrind</pre>

'''Usage:'''
To simulate your program and gather information about it using a specific tool, valgrind can be called like this
<pre>$ valgrind --tool=<tool> --log-file=<logfile> ./example</pre>
For example to check for race conditions the command would look like this:
<pre>$ valgrind --tool=drd --log-file=log.mem ./example</pre>

Beware that the simulation via valgrind can take much longer and can consume much more memory compared to a normal execution.
Furthermore the tools can give false positives, i.e. they report errors even though the are none.

'''Example for helgrind and drd:'''

This small program demonstrates how to detect data races in parallel programs. Both helgrind and drd support POSIX Threads (Pthreads) but not OpenMP so prepare to get many false positives when using these tools with OpenMP.

<source lang=c>
/* pthreads.c */
#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
#include <math.h>

#define NUM_THREADS 2

int v = 0;

void* f(void* x)
{
int j = 0;
int k = *(int*)x;

for(j = 0; j < 100; j++)
v += sin(j) + 1 + k;

return NULL;
}

int main(int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int idx[NUM_THREADS];
int t;

for(t=0; t<NUM_THREADS; t++) {
idx[t] = t;
pthread_create(&threads[t], NULL, f, (void*)&idx[t]);
}

for(t=0; t<NUM_THREADS; t++)
pthread_join(threads[t], NULL);

printf("%i\n", v);

pthread_exit(NULL);

return 0;
}
</source>
The program can be compiled with
<pre>$ gcc -Wall pthreads.c -o pthreads -pthread -lm -O2</pre>
where -pthread tells the compiler to link against the Pthreads library and -lm is needed for linking against the math library.
This program has a race condition, consecutive executions can give different outputs. Such errors are hard to find because they can show up very rarely.
Helgrind can tell you the problematic part of your code:
<pre>
$ valgrind --tool=helgrind ./pthreads
...
==29237== ----------------------------------------------------------------
==29237==
==29237== Possible data race during read of size 4 at 0x600C78 by thread #3
==29237== Locks held: none
==29237== at 0x4007BC: f(void*) (in /pfs/data2/home/xx/xxxx/xxxx/pthreads)
==29237== by 0x4A0C0D4: mythread_wrapper (hg_intercepts.c:219)
==29237== by 0x3EDA6079D0: start_thread (in /lib64/libpthread-2.12.so)
==29237== by 0x3ED9AE8B6C: clone (in /lib64/libc-2.12.so)
==29237==
==29237== This conflicts with a previous write of size 4 by thread #2
==29237== Locks held: none
==29237== at 0x4007FB: f(void*) (in /pfs/data2/home/xx/xxxx/xxxx/pthreads)
==29237== by 0x4A0C0D4: mythread_wrapper (hg_intercepts.c:219)
==29237== by 0x3EDA6079D0: start_thread (in /lib64/libpthread-2.12.so)
==29237== by 0x3ED9AE8B6C: clone (in /lib64/libc-2.12.so)
==29237==
==29237== ----------------------------------------------------------------
...
</pre>
Now we know that we have to check the function f for a possible data race. A similar output can be achieved using drd with the following command:
<pre>$ valgrind --tool=drd ./pthreads</pre>

Valgrind can give even better hints when compiled with debug information (helgrind produces similar output):
<pre>
$ gcc -g -Wall pthreads.c -o pthreads -pthread -lm
$ valgrind --tool=drd ./pthreads
...
==604== Thread 3:
==604== Conflicting load by thread 3 at 0x00600cc0 size 4
==604== at 0x400749: f(void*) (pthreads.c:17)
==604== by 0x4A128B4: vgDrd_thread_wrapper (drd_pthread_intercepts.c:355)
==604== by 0x3EDA6079D0: start_thread (in /lib64/libpthread-2.12.so)
==604== by 0x3ED9AE8B6C: clone (in /lib64/libc-2.12.so)
==604== Allocation context: BSS section of fs/data2/home/xx/xxxx/xxxx/pthreads
==604== Other segment start (thread 2)
==604== (thread finished, call stack no longer available)
==604== Other segment end (thread 2)
==604== (thread finished, call stack no longer available)
...
</pre>
The execution can take longer but now we know that the problem is most likely in line 17 (which is correct).

= Parallel Debugger ddt =

{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #e7aa01;"
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #e7aa01; text-align:left; color:#000; padding:0.2em 0.4em;">Currently only for employees of KIT</div>
|-
| style="color:#000;" | <div style="padding:2px 5px">On bwUniCluster the GUI based distributed debugging tool (ddt) may be used to debug serial as
well as parallel applications. For serial applications also the GNU gdb or Intel idb debugger
may be used. The Intel idb comes with the compiler and information on this tool is available
together with the compiler documentation. In order to debug your program it must be
compiled and linked using the -g compiler option. This will force the compiler to add additional information to the object code which is used by the debugger at runtime.
</div>
|}

ddt consists of a graphical frontend and a backend serial debugger which controls the
application program. One instance of the serial debugger controls one MPI process. Via the
frontend the user interacts with the debugger to select the program that will be debugged,
to specify different options and to monitor the execution of the program. Debugging
commands may be sent to one, all or a subset of the MPI processes.

Before the parallel debugger ddt can be used, it is necessary to load the corresponding
module file:
<pre>
$ module use /opt/bwhpc/ka/modulefiles (only available for employees of KIT)
$ module add debugger/ddt
</pre>

Now ddt may be started with the command
<pre>
$ ddt program
</pre>

where program is the name of your program that you want to debug.

[[File:ddt1_750.jpg]]

Figure: DDT startup window

The above figure shows ddt’s startup window. Before actually starting the debugging session
you should check the contents of several fields in this window:

1. The top line shows the executable file that will be run under control of the debugger. In
the following lines you may input some options that are passed to your program or to the
MPI environment.

2. If your program reads data from stdin you can specify an input file in the startup window.

3. Before starting an MPI program you should check that "Open MPI (Compatability)" or
"Intel MPI" is the MPI implementation that has been selected. If this is not the case, you
have to change this. Otherwise ddt may not be able to run your program. In order to debug
serial programs, the selected MPI implementation should be "none". You may also change
the underlying serial debugger using the "change" button. By default ddt uses its own serial debugger, but it may also use the Intel idb debugger.

4. Select the number of MPI processes that will be started by ddt. If you are using ddt within
a batch job, replace mpirun by ddt in the command line of ????? and make sure that the
chosen number of MPI processes is identical to the number of MPI tasks (-p option ???) that
you selected with the ?????? command. When you debug a serial program, select 1.

5. After you have checked all inputs in the ddt startup window, you can start the debugging
session by pressing the "run" button.

The ddt window now shows the source code of the program that is being debugged and breakpoints can be set by just pointing to the corresponding line and pressing the right
mouse button. So you may step through your program, display the values of variables
and arrays and look at the message queues.

[[File:ddt2_750.jpg]]

Development/GDB

2014-03-04T13:13:52Z

R Doros: Table Header included

{| style="border-style: solid; border-width: 1px"
! Navigation: [[BwHPC_Best_Practices_Repository|bwHPC BPR]]
|}


{|{{Softwarebox}}
|-
! colspan="2" style="text-align:center" | GDB: The GNU Project Debugger
|-
| module load
| devel/gdb

|-
| License
| [http://www.gnu.org/licenses/ GPL]
|-
|Citing
|
|-
| Links
| [http://www.gnu.org/software/gdb/ Homepage]; [http://www.gnu.org/software/gdb/documentation/ Documentation]; [https://sourceware.org/gdb/wiki/ Wiki]; [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists]
|
|-
| Graphical Interface
| No 
|-
| Included in module
| 
|}

= Debugging =

The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.

== Loading ==
There is no need for loading GDB because it is available by default. If you want to use IDB load the Intel compiler module:
<pre>$ module load compiler/intel</pre>

== Documentation ==
'''Online documentation:''' [http://www.gnu.org/software/gdb/documentation/ GDB documentation] and [http://software.intel.com/sites/products/documentation/doclib/stdxe/2013/composerxe/debugger/user_guide/index.htm IDB Documentation]

'''Local documentation:'''
For detailed lists of the different program options consult the man page:
<pre>$ man gdb</pre>
or
<pre>$ man idb</pre>

== Basic commands ==
The code you want to debug should be compiled with the -g option and it is recommended that optimization flags are not set. To start a debug session for a program execute GDB with the program path as parameter:
<pre>$ gdb ./example</pre>

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

{| {{Table}}
|-
! Command
! Description
|-
| help ''cmd''
| Show help for command ''cmd''.
|-
| break ''func''
| Set a breakpoint at function ''func''.
|-
| run
| Start program.
|-
| next
| Go to next program line. Do not enter functions.
|-
| step
| Go to next program line. Enter functions.
|-
| list
| Show the surrounding source code of the currently processed line.
|-
| print ''expr''
| Print the value of the expression ''expr''.
|-
| display ''expr''
| Display the value of the expression ''expr'' every time the program stops.
|-
| watch ''expr''
| Stop when value of the expression ''expr'' changes.
|-
| continue
| Continue execution until a breakpoint or a watchpoint appears.
|-
| backtrace
| Print a list of functions that are currently active.
|-
| quit
| Exit GDB.
|}

'''Example:'''
We debug the following program called bug.c which crashes on execution.
<source lang="c">
#include <stdio.h>

int global = 0;

void begin() {
global = 1;
}

void loop() {
int v[2];
int i, k;

for(i = 0; i < 8; i++) {
k = i/2*2; /* should have been k = i/(2*2); */
v[k] = i;
}
}

void end() {
global = 2;
}

int main() {
begin();
loop();
end();

return 0;
}
</source>

'''Sample GDB session:'''
<pre>
$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27 loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26 begin();
(gdb) next
27 loop();
(gdb) step
loop () at bug.c:13
13 for(i = 0; i < 8; i++)
(gdb) next
15 k = i/2*2;
(gdb) next
16 v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16 v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit
</pre>

== Core dumps ==
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
<pre>$ ulimit -c unlimited</pre>
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
<pre>$ gdb ./ex core.xxx</pre>
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.

== Multithreaded debugging ==
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step.
The following commands are useful for multithreaded debugging:

{| {{Table}}
|-
! Command
! Description
|-
| info threads
| Shows the status of all existing threads.
|-
| thread ''num''
| Switches to the thread with the number ''num''
|}

= Valgrind =

Valgrind can be used for debugging and profiling programs. It ships with various tools, some of them are described in the following table.

{| {{Table}}
|-
! Tool
! Description
|-
| memcheck
| Detects heap array overruns, memory leaks and incorrect freeing of heap memory.
|-
| exp-sgcheck
| Detects stack and global array overruns.
|-
| callgrind
| Profiler. Helps finding bottlenecks.
|-
| helgrind
| Detects race conditions and deadlocks.
|-
| drd
| Detects race conditions. Does not detect deadlocks but needs less memory than helgrind.
|}

'''Loading:''' There is no need for loading valgrind because it is available by default.

'''Online documentation:''' http://valgrind.org/docs/
'''Local documentation:''' <pre>man valgrind</pre>

'''Usage:'''
To simulate your program and gather information about it using a specific tool, valgrind can be called like this
<pre>$ valgrind --tool=<tool> --log-file=<logfile> ./example</pre>
For example to check for race conditions the command would look like this:
<pre>$ valgrind --tool=drd --log-file=log.mem ./example</pre>

Beware that the simulation via valgrind can take much longer and can consume much more memory compared to a normal execution.
Furthermore the tools can give false positives, i.e. they report errors even though the are none.

'''Example for helgrind and drd:'''

This small program demonstrates how to detect data races in parallel programs. Both helgrind and drd support POSIX Threads (Pthreads) but not OpenMP so prepare to get many false positives when using these tools with OpenMP.

<source lang=c>
/* pthreads.c */
#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
#include <math.h>

#define NUM_THREADS 2

int v = 0;

void* f(void* x)
{
int j = 0;
int k = *(int*)x;

for(j = 0; j < 100; j++)
v += sin(j) + 1 + k;

return NULL;
}

int main(int argc, char *argv[])
{
pthread_t threads[NUM_THREADS];
int idx[NUM_THREADS];
int t;

for(t=0; t<NUM_THREADS; t++) {
idx[t] = t;
pthread_create(&threads[t], NULL, f, (void*)&idx[t]);
}

for(t=0; t<NUM_THREADS; t++)
pthread_join(threads[t], NULL);

printf("%i\n", v);

pthread_exit(NULL);

return 0;
}
</source>
The program can be compiled with
<pre>$ gcc -Wall pthreads.c -o pthreads -pthread -lm -O2</pre>
where -pthread tells the compiler to link against the Pthreads library and -lm is needed for linking against the math library.
This program has a race condition, consecutive executions can give different outputs. Such errors are hard to find because they can show up very rarely.
Helgrind can tell you the problematic part of your code:
<pre>
$ valgrind --tool=helgrind ./pthreads
...
==29237== ----------------------------------------------------------------
==29237==
==29237== Possible data race during read of size 4 at 0x600C78 by thread #3
==29237== Locks held: none
==29237== at 0x4007BC: f(void*) (in /pfs/data2/home/xx/xxxx/xxxx/pthreads)
==29237== by 0x4A0C0D4: mythread_wrapper (hg_intercepts.c:219)
==29237== by 0x3EDA6079D0: start_thread (in /lib64/libpthread-2.12.so)
==29237== by 0x3ED9AE8B6C: clone (in /lib64/libc-2.12.so)
==29237==
==29237== This conflicts with a previous write of size 4 by thread #2
==29237== Locks held: none
==29237== at 0x4007FB: f(void*) (in /pfs/data2/home/xx/xxxx/xxxx/pthreads)
==29237== by 0x4A0C0D4: mythread_wrapper (hg_intercepts.c:219)
==29237== by 0x3EDA6079D0: start_thread (in /lib64/libpthread-2.12.so)
==29237== by 0x3ED9AE8B6C: clone (in /lib64/libc-2.12.so)
==29237==
==29237== ----------------------------------------------------------------
...
</pre>
Now we know that we have to check the function f for a possible data race. A similar output can be achieved using drd with the following command:
<pre>$ valgrind --tool=drd ./pthreads</pre>

Valgrind can give even better hints when compiled with debug information (helgrind produces similar output):
<pre>
$ gcc -g -Wall pthreads.c -o pthreads -pthread -lm
$ valgrind --tool=drd ./pthreads
...
==604== Thread 3:
==604== Conflicting load by thread 3 at 0x00600cc0 size 4
==604== at 0x400749: f(void*) (pthreads.c:17)
==604== by 0x4A128B4: vgDrd_thread_wrapper (drd_pthread_intercepts.c:355)
==604== by 0x3EDA6079D0: start_thread (in /lib64/libpthread-2.12.so)
==604== by 0x3ED9AE8B6C: clone (in /lib64/libc-2.12.so)
==604== Allocation context: BSS section of fs/data2/home/xx/xxxx/xxxx/pthreads
==604== Other segment start (thread 2)
==604== (thread finished, call stack no longer available)
==604== Other segment end (thread 2)
==604== (thread finished, call stack no longer available)
...
</pre>
The execution can take longer but now we know that the problem is most likely in line 17 (which is correct).

= Parallel Debugger ddt =

{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #e7aa01;"
| style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #e7aa01; text-align:left; color:#000; padding:0.2em 0.4em;">Currently only for employees of KIT</div>
|-
| style="color:#000;" | <div style="padding:2px 5px">On bwUniCluster the GUI based distributed debugging tool (ddt) may be used to debug serial as
well as parallel applications. For serial applications also the GNU gdb or Intel idb debugger
may be used. The Intel idb comes with the compiler and information on this tool is available
together with the compiler documentation. In order to debug your program it must be
compiled and linked using the -g compiler option. This will force the compiler to add additional information to the object code which is used by the debugger at runtime.
</div>
|}

ddt consists of a graphical frontend and a backend serial debugger which controls the
application program. One instance of the serial debugger controls one MPI process. Via the
frontend the user interacts with the debugger to select the program that will be debugged,
to specify different options and to monitor the execution of the program. Debugging
commands may be sent to one, all or a subset of the MPI processes.

Before the parallel debugger ddt can be used, it is necessary to load the corresponding
module file:
<pre>
$ module use /opt/bwhpc/ka/modulefiles (only available for employees of KIT)
$ module add debugger/ddt
</pre>

Now ddt may be started with the command
<pre>
$ ddt program
</pre>

where program is the name of your program that you want to debug.

[[File:ddt1_750.jpg]]

Figure: DDT startup window

The above figure shows ddt’s startup window. Before actually starting the debugging session
you should check the contents of several fields in this window:

1. The top line shows the executable file that will be run under control of the debugger. In
the following lines you may input some options that are passed to your program or to the
MPI environment.

2. If your program reads data from stdin you can specify an input file in the startup window.

3. Before starting an MPI program you should check that "Open MPI (Compatability)" or
"Intel MPI" is the MPI implementation that has been selected. If this is not the case, you
have to change this. Otherwise ddt may not be able to run your program. In order to debug
serial programs, the selected MPI implementation should be "none". You may also change
the underlying serial debugger using the "change" button. By default ddt uses its own serial debugger, but it may also use the Intel idb debugger.

4. Select the number of MPI processes that will be started by ddt. If you are using ddt within
a batch job, replace mpirun by ddt in the command line of ????? and make sure that the
chosen number of MPI processes is identical to the number of MPI tasks (-p option ???) that
you selected with the ?????? command. When you debug a serial program, select 1.

5. After you have checked all inputs in the ddt startup window, you can start the debugging
session by pressing the "run" button.

The ddt window now shows the source code of the program that is being debugged and breakpoints can be set by just pointing to the corresponding line and pressing the right
mouse button. So you may step through your program, display the values of variables
and arrays and look at the message queues.

[[File:ddt2_750.jpg]]