Difference between revisions of "Development/GDB"
S Richling (talk | contribs) (→Branch record tracing) |
|||
| (48 intermediate revisions by 7 users not shown) | |||
| Line 1: | Line 1: | ||
| + | {{Softwarepage|devel/gdb}} |
||
| − | {| style="border-style: solid; border-width: 1px" |
||
| + | |||
| − | ! Navigation: [[BwHPC_Best_Practices_Repository|bwHPC BPR]] |
||
| + | {| width=600px class="wikitable" |
||
| + | |- |
||
| + | ! Description !! Content |
||
| + | |- |
||
| + | | module load |
||
| + | | devel/gdb |
||
| + | |- |
||
| + | | License |
||
| + | | [http://www.gnu.org/licenses/ GPL] |
||
| + | |- |
||
| + | |Citing |
||
| + | | n/a |
||
| + | |- |
||
| + | | Links |
||
| + | | [http://www.gnu.org/software/gdb/ Homepage] | [http://www.gnu.org/software/gdb/documentation/ Documentation] | [https://sourceware.org/gdb/wiki/ Wiki] | [http://www.gnu.org/software/gdb/mailing-lists/ Mailinglists] |
||
| + | |- |
||
| + | | Graphical Interface |
||
| + | | No |
||
| + | |- |
||
| + | | Included modules |
||
| + | | icc | icpc | ifort | idb |
||
|} |
|} |
||
| + | <br> |
||
| + | = Description = |
||
| + | The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. In the past Intel supported their own '''idb''' debugger, however this has been deprecated in favor of their own port called <kbd>gdb-ia</kbd>. |
||
| + | <br> |
||
| − | = |
+ | = Basic commands = |
| + | The code you want to debug should be compiled with the <kbd><font color=green>-g</font></kbd> option. If the optimization flag is not set, GCC will still do some basic optimization, like dead-code elimination or reorder instruction execution obfuscating the order when debugging. Therefore, it is recommended to turn off optimization explicitly with the <font color=green>-O0</font> parameter for debugging. To start a debug session for a program execute GDB with the program path as parameter: |
||
| − | |||
| − | The '''GNU Debugger (GDB)''' is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. The '''Intel Debugger (IDB)''' uses the same commands for basic debugging as GDB and hence can be used instead of GDB just by substituting idbc for gdb. |
||
| − | |||
| − | == Loading == |
||
| − | There is no need for loading GDB because it is available by default. If you want to use IDB load the Intel compiler module: |
||
| − | <pre>$ module load compiler/intel</pre> |
||
| − | |||
| − | == Documentation == |
||
| − | '''Online documentation:''' [http://www.gnu.org/software/gdb/documentation/ GDB documentation] and [http://software.intel.com/sites/products/documentation/doclib/stdxe/2013/composerxe/debugger/user_guide/index.htm IDB Documentation] |
||
| − | |||
| − | '''Local documentation:''' |
||
| − | For detailed lists of the different program options consult the man page: |
||
| − | <pre>$ man gdb</pre> |
||
| − | or |
||
| − | <pre>$ man idb</pre> |
||
| − | |||
| − | == Basic commands == |
||
| − | The code you want to debug should be compiled with the -g option and it is recommended that optimization flags are not set. To start a debug session for a program execute GDB with the program path as parameter: |
||
<pre>$ gdb ./example</pre> |
<pre>$ gdb ./example</pre> |
||
Inside GDB is a prompt where you can enter commands. Important commands are listed below. |
Inside GDB is a prompt where you can enter commands. Important commands are listed below. |
||
| + | {| width=600px class="wikitable" |
||
| − | {| {{Table}} |
||
|- |
|- |
||
! Command |
! Command |
||
| Line 60: | Line 68: | ||
| continue |
| continue |
||
| Continue execution until a breakpoint or a watchpoint appears. |
| Continue execution until a breakpoint or a watchpoint appears. |
||
| + | |- |
||
| + | | backtrace |
||
| + | | Print a list of functions that are currently active. |
||
|- |
|- |
||
| quit |
| quit |
||
| Line 65: | Line 76: | ||
|} |
|} |
||
| − | '''Example:''' |
||
| − | We debug the following program called bug.c which crashes on execution. |
||
| − | <source lang="c"> |
||
| − | #include <stdio.h> |
||
| − | int global = 0; |
||
| + | = Branch record tracing = |
||
| − | void begin() { |
||
| + | Starting with GBD-10.1, the debugger has been installed with Intel Processor Trace [https://github.com/intel/libipt libipt], allowing recording and replaying of process state. |
||
| − | global = 1; |
||
| + | This allows disassembling previously executed instructions, checking for previously called functions and branch tracing. |
||
| − | } |
||
| + | Honestly, Segmentation Violations are better caught using [[Development/Valgrind|Valgrind]]. However in this case, |
||
| − | void loop() { |
||
| + | <kbd>valgrind</kbd> would ''not'' have helped: this loops overwrites <kbd>v</kbd> an |
||
| − | int v[2]; |
||
| + | array of 2 ints on the stack and the return address leading to the execution of IP <kbd>0x07</kbd>. |
||
| − | int i, k; |
||
| + | More information is available in [https://sourceware.org/gdb/current/onlinedocs/gdb/Process-Record-and-Replay.html#Process-Record-and-Replay gdb's feature documentation] |
||
| − | for(i = 0; i < 8; i++) { |
||
| − | k = i/2*2; /* should have been k = i/(2*2); */ |
||
| − | v[k] = i; |
||
| − | } |
||
| − | } |
||
| + | <br> |
||
| − | void end() { |
||
| − | global = 2; |
||
| − | } |
||
| + | = Core dumps = |
||
| − | int main() { |
||
| + | When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example: |
||
| − | begin(); |
||
| + | <pre>$ ulimit -c unlimited</pre> |
||
| − | loop(); |
||
| + | Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex): |
||
| − | end(); |
||
| + | <pre>$ gdb ./ex core.xxx</pre> |
||
| + | Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error. |
||
| + | <br> |
||
| − | return 0; |
||
| − | } |
||
| − | </source> |
||
| + | = Multithreaded debugging = |
||
| − | '''Sample GDB session:''' |
||
| − | <pre> |
||
| − | $ gcc -g bug.c -o bug |
||
| − | $ gdb ./bug |
||
| − | GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1) |
||
| − | Copyright (C) 2010 Free Software Foundation, Inc. |
||
| − | License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> |
||
| − | This is free software: you are free to change and redistribute it. |
||
| − | There is NO WARRANTY, to the extent permitted by law. Type "show copying" |
||
| − | and "show warranty" for details. |
||
| − | This GDB was configured as "x86_64-redhat-linux-gnu". |
||
| − | For bug reporting instructions, please see: |
||
| − | <http://www.gnu.org/software/gdb/bugs/>... |
||
| − | Reading symbols from /pfs/data2/home/xx/xxx/xxxx/bug...done. |
||
| − | (gdb) break main |
||
| − | Breakpoint 1 at 0x4005b2: file bug.c, line 26. |
||
| − | (gdb) run |
||
| − | Starting program: /pfs/data2/home/xx/xxx/xxxx/bug |
||
| − | |||
| − | Breakpoint 1, main () at bug.c:26 |
||
| − | 26 begin(); |
||
| − | Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64 |
||
| − | (gdb) next |
||
| − | 27 loop(); |
||
| − | (gdb) next |
||
| − | |||
| − | Program received signal SIGSEGV, Segmentation fault. |
||
| − | 0x0000000000000005 in ?? () |
||
| − | (gdb) # now we know that the bug is in loop(). start again. |
||
| − | (gdb) run |
||
| − | The program being debugged has been started already. |
||
| − | Start it from the beginning? (y or n) y |
||
| − | Starting program: /pfs/data2/home/xx/xxx/xxxx/bug |
||
| − | |||
| − | Breakpoint 1, main () at bug.c:26 |
||
| − | 26 begin(); |
||
| − | (gdb) next |
||
| − | 27 loop(); |
||
| − | (gdb) step |
||
| − | loop () at bug.c:13 |
||
| − | 13 for(i = 0; i < 8; i++) |
||
| − | (gdb) next |
||
| − | 15 k = i/2*2; |
||
| − | (gdb) next |
||
| − | 16 v[k] = i; |
||
| − | (gdb) # maybe k gets too big? |
||
| − | (gdb) watch (k >= 2) |
||
| − | Hardware watchpoint 2: (k >= 2) |
||
| − | (gdb) continue |
||
| − | Continuing. |
||
| − | Hardware watchpoint 2: (k >= 2) |
||
| − | |||
| − | Old value = 0 |
||
| − | New value = 1 |
||
| − | loop () at bug.c:16 |
||
| − | 16 v[k] = i; |
||
| − | (gdb) # k is too big |
||
| − | (gdb) print k |
||
| − | $1 = 2 |
||
| − | (gdb) print i |
||
| − | $2 = 2 |
||
| − | (gdb) quit |
||
| − | </pre> |
||
| − | |||
| − | == Multithreaded debugging == |
||
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step. |
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step. |
||
The following commands are useful for multithreaded debugging: |
The following commands are useful for multithreaded debugging: |
||
| − | |||
| − | {| {{Table}} |
||
| − | |- |
||
| − | ! Command |
||
| − | ! Description |
||
| − | |- |
||
| − | | info threads |
||
| − | | Shows the status of all existing threads. |
||
| − | |- |
||
| − | | thread ''num'' |
||
| − | | Switches to the thread with the number ''num'' |
||
| − | |} |
||
| − | |||
| − | == Distributed debugging == |
||
| − | It is also possible to debug MPI applications with GDB but only with a handful of processes. You can start the debugger for every process separately and use the usual commands to control the execution of each process. |
||
| − | To show how to start the debugging session we use a program from the examples directory of the Intel MPI (see $MPI_EXA_DIR after loading the module) called pi3.c. First we need to load Intel MPI (see [[BwUniCluster_Environment_Modules | environment modules]]) and compile the code with debug information |
||
| − | <pre>$ mpicc -g pi3.c -o pi3</pre> |
||
| − | Next we can submit a job which starts 2 instances of GDB. Therefore you need to create a jobscript (see [[BwUniCluster_Batch_Jobs | batch jobs]]) called job.sh with the following commands: |
||
| − | <source lang="bash"> |
||
| − | #!/bin/sh |
||
| − | module load mpi/impi |
||
| − | mpirun -n 2 xterm -e gdb ./pi3 |
||
| − | </source> |
||
| − | We need to pass the DISPLAY variable and specify the multinode queue when submitting the job: |
||
| − | <pre>$ msub -N mpi_dbg -l nodes=2:ppn=1 -q multinode -v DISPLAY ./job.sh</pre> |
||
| − | Now two xterm windows with gdb started in them should appear and you can start debugging. |
||
| − | |||
| − | |||
| − | = Parallel Debugger ddt = |
||
| − | |||
| − | {| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #e7aa01;" |
||
| − | | style="padding:2px;" | <div style="margin:3px; background:#cef2e0; font-size:120%; font-weight:bold; border:1px solid #e7aa01; text-align:left; color:#000; padding:0.2em 0.4em;">Currently only for employees of KIT</div> |
||
| − | |- |
||
| − | | style="color:#000;" | <div style="padding:2px 5px">On bwUniCluster the GUI based distributed debugging tool (ddt) may be used to debug serial as |
||
| − | well as parallel applications. For serial applications also the GNU gdb or Intel idb debugger |
||
| − | may be used. The Intel idb comes with the compiler and information on this tool is available |
||
| − | together with the compiler documentation. In order to debug your program it must be |
||
| − | compiled and linked using the -g compiler option. This will force the compiler to add additional information to the object code which is used by the debugger at runtime. |
||
| − | </div> |
||
| − | |} |
||
| − | |||
| − | |||
| − | |||
| − | ddt consists of a graphical frontend and a backend serial debugger which controls the |
||
| − | application program. One instance of the serial debugger controls one MPI process. Via the |
||
| − | frontend the user interacts with the debugger to select the program that will be debugged, |
||
| − | to specify different options and to monitor the execution of the program. Debugging |
||
| − | commands may be sent to one, all or a subset of the MPI processes. |
||
| − | |||
| − | Before the parallel debugger ddt can be used, it is necessary to load the corresponding |
||
| − | module file: |
||
| − | <pre> |
||
| − | $ module use /opt/bwhpc/ka/modulefiles (only available for employees of KIT) |
||
| − | $ module add debugger/ddt |
||
| − | </pre> |
||
| − | |||
| − | Now ddt may be started with the command |
||
| − | <pre> |
||
| − | $ ddt program |
||
| − | </pre> |
||
| − | |||
| − | where program is the name of your program that you want to debug. |
||
| − | |||
| − | [[File:ddt1_750.jpg]] |
||
| − | |||
| − | Figure: DDT startup window |
||
| − | |||
| − | The above figure shows ddt’s startup window. Before actually starting the debugging session |
||
| − | you should check the contents of several fields in this window: |
||
| − | |||
| − | 1. The top line shows the executable file that will be run under control of the debugger. In |
||
| − | the following lines you may input some options that are passed to your program or to the |
||
| − | MPI environment. |
||
| − | |||
| − | 2. If your program reads data from stdin you can specify an input file in the startup window. |
||
| − | |||
| − | 3. Before starting an MPI program you should check that "Open MPI (Compatability)" or |
||
| − | "Intel MPI" is the MPI implementation that has been selected. If this is not the case, you |
||
| − | have to change this. Otherwise ddt may not be able to run your program. In order to debug |
||
| − | serial programs, the selected MPI implementation should be "none". You may also change |
||
| − | the underlying serial debugger using the "change" button. By default ddt uses its own serial debugger, but it may also use the Intel idb debugger. |
||
| − | |||
| − | 4. Select the number of MPI processes that will be started by ddt. If you are using ddt within |
||
| − | a batch job, replace mpirun by ddt in the command line of ????? and make sure that the |
||
| − | chosen number of MPI processes is identical to the number of MPI tasks (-p option ???) that |
||
| − | you selected with the ?????? command. When you debug a serial program, select 1. |
||
| − | |||
| − | 5. After you have checked all inputs in the ddt startup window, you can start the debugging |
||
| − | session by pressing the "run" button. |
||
| − | |||
| − | |||
| − | The ddt window now shows the source code of the program that is being debugged and breakpoints can be set by just pointing to the corresponding line and pressing the right |
||
| − | mouse button. So you may step through your program, display the values of variables |
||
| − | and arrays and look at the message queues. |
||
| − | |||
| − | [[File:ddt2_750.jpg]] |
||
Latest revision as of 00:49, 9 December 2022
|
The main documentation is available via |
| Description | Content |
|---|---|
| module load | devel/gdb |
| License | GPL |
| Citing | n/a |
| Links | Homepage | Documentation | Wiki | Mailinglists |
| Graphical Interface | No |
| Included modules | icc | icpc | ifort | idb |
Contents
1 Description
The GNU Debugger (GDB) is a standard debugger for serial programs although it can be used for parallel and even distributed programs with few processes too. In the past Intel supported their own idb debugger, however this has been deprecated in favor of their own port called gdb-ia.
2 Basic commands
The code you want to debug should be compiled with the -g option. If the optimization flag is not set, GCC will still do some basic optimization, like dead-code elimination or reorder instruction execution obfuscating the order when debugging. Therefore, it is recommended to turn off optimization explicitly with the -O0 parameter for debugging. To start a debug session for a program execute GDB with the program path as parameter:
$ gdb ./example
Inside GDB is a prompt where you can enter commands. Important commands are listed below.
| Command | Description |
|---|---|
| help cmd | Show help for command cmd. |
| break func | Set a breakpoint at function func. |
| run | Start program. |
| next | Go to next program line. Do not enter functions. |
| step | Go to next program line. Enter functions. |
| list | Show the surrounding source code of the currently processed line. |
| print expr | Print the value of the expression expr. |
| display expr | Display the value of the expression expr every time the program stops. |
| watch expr | Stop when value of the expression expr changes. |
| continue | Continue execution until a breakpoint or a watchpoint appears. |
| backtrace | Print a list of functions that are currently active. |
| quit | Exit GDB. |
3 Branch record tracing
Starting with GBD-10.1, the debugger has been installed with Intel Processor Trace libipt, allowing recording and replaying of process state. This allows disassembling previously executed instructions, checking for previously called functions and branch tracing.
Honestly, Segmentation Violations are better caught using Valgrind. However in this case, valgrind would not have helped: this loops overwrites v an array of 2 ints on the stack and the return address leading to the execution of IP 0x07.
More information is available in gdb's feature documentation
4 Core dumps
When the program crashes, a log file (called core dump) can be created which contains the state of the program when it crashed. This is turned off by default because these core dumps can get quite large. If you want to turn it on you have to change your ulimits, for example:
$ ulimit -c unlimited
Every time your program crashes a new file called core.xxx (where xxx is a number) will be created in the directory from which you started the executable. You can call gdb to examine your core dump using the following command (assuming your program is called ex):
$ gdb ./ex core.xxx
Now you can print a backtrace to check in which function the error happened and what values the parameters had. Additionally you can examine the values of your variables to reproduce the error.
5 Multithreaded debugging
GDB can also be useful for multithreaded applications for example when OpenMP was used. By going through each thread separately you can better see what is really going on and you can check the computation step by step. The following commands are useful for multithreaded debugging:
