Development/GDB

Navigation: bwHPC BPR

Debugging

The GNU Debugger (GDB) is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The Intel Debugger (IDB) uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb.

Loading

There is no need for loading GDB because it is available by default. If you want to use IDB load the Intel compiler module:

$ module load compiler/intel

Documentation

Online documentation: GDB documentation and IDB Documentation

Local documentation: For detailed lists of the different program options consult the man page:

$ man gdb

or

$ man idb

Basic commands

The code you want to debug should be compiled with the -g option and it is recommended that optimization flags are not set. To start a debug session for a program execute GDB with the program path as parameter:

$ gdb ./example

Inside GDB is a prompt where you can enter commands. Important commands are listed below.

Command	Description
help cmd	Show help for command cmd.
break func	Set a breakpoint at function func.
run	Start program.
next	Go to next program line. Do not enter functions.
step	Go to next program line. Enter functions.
list	Show the surrounding source code of the currently processed line.
print expr	Print the value of the expression expr.
display expr	Display the value of the expression expr every time the program stops.
watch expr	Stop when value of the expression expr changes.
continue	Continue execution until a breakpoint or a watchpoint appears.
backtrace	Print a list of functions that are currently active.
quit	Exit GDB.

Example: We debug the following program called bug.c which crashes on execution.

#include <stdio.h>

int global = 0;

void begin() {
    global = 1;
}

void loop() {
    int v[2];
    int i, k;

    for(i = 0; i < 8; i++) {
        k = i/2*2;	/* should have been k = i/(2*2); */
        v[k] = i;
    }
}

void end() {
    global = 2;
}

int main() {
    begin();
    loop();
    end();

    return 0;
}

Sample GDB session:

$ gcc -g bug.c -o bug
$ gdb ./bug
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done.
(gdb) break main
Breakpoint 1 at 0x4005b2: file bug.c, line 26.
(gdb) run
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26              begin();
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64
(gdb) next
27              loop();
(gdb) next

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000005 in ?? ()
(gdb) # now we know that the bug is in loop(). start again.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /pfs/data2/home/xx/xxx/xxxx/bug

Breakpoint 1, main () at bug.c:26
26              begin();
(gdb) next
27              loop();
(gdb) step
loop () at bug.c:13
13              for(i = 0; i < 8; i++)
(gdb) next
15                      k = i/2*2;
(gdb) next
16                      v[k] = i;
(gdb) # maybe k gets too big?
(gdb) watch (k >= 2)
Hardware watchpoint 2: (k >= 2)
(gdb) continue
Continuing.
Hardware watchpoint 2: (k >= 2)

Old value = 0
New value = 1
loop () at bug.c:16
16                      v[k] = i;
(gdb) # k is too big
(gdb) print k
$1 = 2
(gdb) print i
$2 = 2
(gdb) quit

Core dumps

When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:

$ ulimit -c unlimited

Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):

$ gdb ./ex core.xxx

Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.

Multithreaded debugging

GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step. The following commands are useful for multithreaded debugging:

Command	Description
info threads	Shows the status of all existing threads.
thread num	Switches to the thread with the number num

Valgrind

Valgrind can be used for debugging and profiling programs. It ships with various tools, some of them are described in the following table.

Tool	Description
memcheck	Detects heap array overruns, memory leaks and incorrect freeing of heap memory.
exp-sgcheck	Detects stack and global array overruns.
callgrind	Profiler. Helps finding bottlenecks.
helgrind	Detects race conditions and deadlocks.
drd	Detects race conditions. Does not detect deadlocks but needs less memory than helgrind.

Loading: There is no need for loading valgrind because it is available by default.

Online documentation: http://valgrind.org/docs/

Local documentation:

man valgrind

Usage: To simulate your program and gather information about it using a specific tool, valgrind can be called like this

$ valgrind --tool=<tool> --log-file=<logfile> ./example

For example to check for race conditions the command would look like this:

$ valgrind --tool=drd --log-file=log.mem ./example

Beware that the simulation via valgrind can take much longer and can consume much more memory compared to a normal execution. Furthermore the tools can give false positives, i.e. they report errors even though the are none.

Example for helgrind and drd:

This small program demonstrates how to detect data races in parallel programs. Both helgrind and drd support POSIX Threads (Pthreads) but not OpenMP so prepare to get many false positives when using these tools with OpenMP.

/* pthreads.c */
#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
#include <math.h>

#define NUM_THREADS 2

int v = 0;

void* f(void* x)
{
    int j = 0;
    int k = *(int*)x;
    
    for(j = 0; j < 100; j++)
        v += sin(j) + 1 + k;

    return NULL;
}

int main(int argc, char *argv[])
{
    pthread_t threads[NUM_THREADS];
    int idx[NUM_THREADS];
    int t;
   
    for(t=0; t<NUM_THREADS; t++) {
        idx[t] = t;
        pthread_create(&threads[t], NULL, f, (void*)&idx[t]);
    }
    
    for(t=0; t<NUM_THREADS; t++)
        pthread_join(threads[t], NULL);

    printf("%i\n", v);

    pthread_exit(NULL);

    return 0;
}

The program can be compiled with

$ gcc -Wall pthreads.c -o pthreads -pthread -lm -O2

where -pthread tells the compiler to link against the Pthreads library and -lm is needed for linking against the math library. This program has a race condition, consecutive executions can give different outputs. Such errors are hard to find because they can show up very rarely. Helgrind can tell you the problematic part of your code:

$ valgrind --tool=helgrind ./pthreads
...
==29237== ----------------------------------------------------------------
==29237== 
==29237== Possible data race during read of size 4 at 0x600C78 by thread #3
==29237== Locks held: none
==29237==    at 0x4007BC: f(void*) (in /pfs/data2/home/xx/xxxx/xxxx/pthreads)
==29237==    by 0x4A0C0D4: mythread_wrapper (hg_intercepts.c:219)
==29237==    by 0x3EDA6079D0: start_thread (in /lib64/libpthread-2.12.so)
==29237==    by 0x3ED9AE8B6C: clone (in /lib64/libc-2.12.so)
==29237== 
==29237== This conflicts with a previous write of size 4 by thread #2
==29237== Locks held: none
==29237==    at 0x4007FB: f(void*) (in /pfs/data2/home/xx/xxxx/xxxx/pthreads)
==29237==    by 0x4A0C0D4: mythread_wrapper (hg_intercepts.c:219)
==29237==    by 0x3EDA6079D0: start_thread (in /lib64/libpthread-2.12.so)
==29237==    by 0x3ED9AE8B6C: clone (in /lib64/libc-2.12.so)
==29237== 
==29237== ----------------------------------------------------------------
...

Now we know that we have to check the function f for a possible data race. A similar output can be achieved using drd with the following command:

$ valgrind --tool=drd ./pthreads

Valgrind can give even better hints when compiled with debug information (helgrind produces similar output):

$ gcc -g -Wall pthreads.c -o pthreads -pthread -lm
$ valgrind --tool=drd ./pthreads
...
==604== Thread 3:
==604== Conflicting load by thread 3 at 0x00600cc0 size 4
==604==    at 0x400749: f(void*) (pthreads.c:17)
==604==    by 0x4A128B4: vgDrd_thread_wrapper (drd_pthread_intercepts.c:355)
==604==    by 0x3EDA6079D0: start_thread (in /lib64/libpthread-2.12.so)
==604==    by 0x3ED9AE8B6C: clone (in /lib64/libc-2.12.so)
==604== Allocation context: BSS section of fs/data2/home/xx/xxxx/xxxx/pthreads
==604== Other segment start (thread 2)
==604==    (thread finished, call stack no longer available)
==604== Other segment end (thread 2)
==604==    (thread finished, call stack no longer available)
...

The execution can take longer but now we know that the problem is most likely in line 17 (which is correct).

Parallel Debugger ddt

Currently only for employees of KIT

On bwUniCluster the GUI based distributed debugging tool (ddt) may be used to debug serial as well as parallel applications. For serial applications also the GNU gdb or Intel idb debugger may be used. The Intel idb comes with the compiler and information on this tool is available together with the compiler documentation. In order to debug your program it must be compiled and linked using the -g compiler option. This will force the compiler to add additional information to the object code which is used by the debugger at runtime.

ddt consists of a graphical frontend and a backend serial debugger which controls the application program. One instance of the serial debugger controls one MPI process. Via the frontend the user interacts with the debugger to select the program that will be debugged, to specify different options and to monitor the execution of the program. Debugging commands may be sent to one, all or a subset of the MPI processes.

Before the parallel debugger ddt can be used, it is necessary to load the corresponding module file:

$ module use /opt/bwhpc/ka/modulefiles      (only available for employees of KIT)
$ module add debugger/ddt   

Now ddt may be started with the command

$ ddt program

where program is the name of your program that you want to debug.

Figure: DDT startup window

The above figure shows ddt’s startup window. Before actually starting the debugging session you should check the contents of several fields in this window:

1. The top line shows the executable file that will be run under control of the debugger. In the following lines you may input some options that are passed to your program or to the MPI environment.

2. If your program reads data from stdin you can specify an input file in the startup window.

3. Before starting an MPI program you should check that "Open MPI (Compatability)" or "Intel MPI" is the MPI implementation that has been selected. If this is not the case, you have to change this. Otherwise ddt may not be able to run your program. In order to debug serial programs, the selected MPI implementation should be "none". You may also change the underlying serial debugger using the "change" button. By default ddt uses its own serial debugger, but it may also use the Intel idb debugger.

4. Select the number of MPI processes that will be started by ddt. If you are using ddt within a batch job, replace mpirun by ddt in the command line of ????? and make sure that the chosen number of MPI processes is identical to the number of MPI tasks (-p option ???) that you selected with the ?????? command. When you debug a serial program, select 1.

5. After you have checked all inputs in the ddt startup window, you can start the debugging session by pressing the "run" button.

The ddt window now shows the source code of the program that is being debugged and breakpoints can be set by just pointing to the corresponding line and pressing the right mouse button. So you may step through your program, display the values of variables and arrays and look at the message queues.