GDB

The GNU Debugger GDB is the most common debug tool for Linux. It features most used features one can think of, including server-client debug architecture (so you run the heavy debug part on your host/pc machine), but lack some bits as checkpoint-restart during execution.

Documentation
GDB ships with extensive documentation at http://www.gnu.org/software/gdb/documentation/, but there are some good quick reference cards as well.

Basic Usage
Documentation is so large that sometimes its hard to get started, so most simple tasks can be done with the following commands, but please read GDB docs as soon as possible!

Debug packages
First note that in order to use GDB in an efficient way,you need the debug packages,that is to say the -dbg version of your package. Under the opkg package management if you installed a binary like that: opkg install binary you need to install the debug part of it: opkg install binary-dbg

Without debug packages you will be limited to assembly debugging(no source level debugging)

Here are some example of how usefull is debug symbols:

Without debug symbols: DISPLAY=:0.0 gdbserver 10.10.10.10:8090 matchbox-desktop #target gnutoo@gnutoo-desktop:~/embedded/arm-2010q1/bin$ ./arm-none-linux-gnueabi-gdb GNU gdb (Sourcery G++ Lite 2010q1-202) 7.0.50.20100218-cvs Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later  This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=i686-pc-linux-gnu --target=arm-none-linux-gnueabi". For bug reporting instructions, please see: . (gdb) target remote 10.10.10.10:8090 Remote debugging using 10.10.10.10:8090 0x400007e0 in ?? (gdb) c Continuing. Cannot access memory at address 0x0 Program received signal SIGSEGV, Segmentation fault. 0x403e1254 in ?? (gdb) bt Cannot access memory at address 0x302e3032 With debugggin symbols: opkg install matchbox-desktop-dbg ... DISPLAY=:0.0 gdbserver 10.10.10.10:8090 matchbox-desktop #target root@gnutoo-desktop:/home/gnutoo/embedded/arm-2010q1/bin# ./arm-none-linux-gnueabi-gdb GNU gdb (Sourcery G++ Lite 2010q1-202) 7.0.50.20100218-cvs Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later  This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=i686-pc-linux-gnu --target=arm-none-linux-gnueabi". For bug reporting instructions, please see: . (gdb) set sysroot /mnt/target/ (gdb) file /mnt/target/usr/bin/matchbox-desktop Reading symbols from /mnt/target/usr/bin/matchbox-desktop...Reading symbols from /mnt/target/usr/bin/.debug/matchbox-desktop...done. done. (gdb) target remote 10.10.10.10:8090 Remote debugging using 10.10.10.10:8090 Reading symbols from /mnt/target/lib/ld-linux.so.3...Reading symbols from /mnt/target/lib/.debug/ld-2.9.so...done. done. Loaded symbols for /mnt/target/lib/ld-linux.so.3 0x400007e0 in _start from /mnt/target/lib/ld-linux.so.3 (gdb) c Continuing. Program received signal SIGSEGV, Segmentation fault. 0x403e1254 in strlen from /mnt/target/lib/libc.so.6 (gdb) bt
 * 1) 0 0x403e1254 in ??
 * 1) 0 0x403e1254 in strlen  from /mnt/target/lib/libc.so.6
 * 2) 1 0x4003350c in mb_pixbuf_img_new_from_file (pb=0x1fd58, filename=0x0) at mbpixbuf.c:1352
 * 3) 2 0x0000c8fc in mbdesktop_set_scroll_buttons (mb=0x1b008) at mbdesktop.c:689
 * 4) 3 0x0000d334 in mbdesktop_init (argc=, argv= ) at mbdesktop.c:1022
 * 5) 4 0x0000e024 in main (argc=0, argv=0x0) at mbdesktop.c:1793

Note that we have lines that looks like: When we have the debug symbols. The lack of some debug libs could result in a mix of lines like the one just shown above, and lines like that: Which means that the later lines don't have debug symbols(something went wrong).
 * 1) 3 0x0000d334 in mbdesktop_init (argc=, argv= ) at mbdesktop.c:1022
 * 1) 1 0x405ce03c in g_object_ref  from /home/gnutoo/embedded/sshfs/usr/lib/libgobject-2.0.so.0

Also Note that if you have restricted space on the target,you could use NFS root if you have an ethernet connection.

No packages
As I don't have packages which segfault at hand,I'll create a binary which segfault First start exporting the path of the toolchain if it's not done yet: $ cd arm-2010q1/bin #codesourcey $ export PATH=$(pwd):$PATH Then create a main.c file with the following content: struct test{ struct test *crash; }; int main{ struct test *test = NULL; test = test->crash; } Then compile it: $ arm-none-linux-gnueabi-gcc -ggdb3 -static -o binary main.c
 * 1) include 
 * ggdb3 is for including as much debug informations as possible
 * -static is only for our test,it's for avoiding runtime issues in case the libc is different or incompatible between codesourcey and our target(in case of ucilbc for instance)

Cross or not crosss
A common error while using gdb,is using the wrong gdb. In order to debug a program that runs on a target from your host you need a cross gdb,that is to say a GDB that runs on your computer but can debug the target architecture. Such version is normally included in your toolchain/SDK or buildable if you use a build system

If you want to debug use GDB on target you can too,it's easier but has a huge drawback(amongs other) : ram usage (you can easily use too much memory and have an OOM(out of memory) )

CodeSourcery Sourcery G++ Lite
Let's say that you downloaded arm-2010q1-202-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2

You will need to unpack it,and to run the following command on your host computer (instead of just running gdb): cd arm-2010q1/bin ./arm-none-linux-gnueabi-gdb

Openembedded
If your build system is also your debug workstation do: bitbake gdb-cross gdbserver And it will build a cross gdb for your host and gdbserver for your target The resulting binaries will be found in your TMPDIR here: /home/embedded/tmpdir/cross/armv6/bin/ replace "/home/embedded/tmpdir" by your tmpdir and armv6 by your target architecture The binary name changes according to your distro settings,for me it was: arm-angstrom-linux-gnueabi-gdb So in order to run it do: ./arm-angstrom-linux-gnueabi-gdb

Note on sources
If your workstation is not your build system,in order to view sources in gdb(else you will only have the path of the source file and the line number) look here

Without the sources path: (gdb) bt (gdb) f 0 (gdb) l 1784	mbdesktop.c: No such file or directory. in mbdesktop.c (gdb) With the sources path set: (gdb) set substitute-path / /media/port4 (gdb) bt (gdb) f 0 (gdb) l 1784	     fprintf(stderr, "matchbox-desktop: failed to load any item modules.\n"); 1785	   } 1786	} 1787	  1788	int 1789	main(int argc, char **argv) 1790	{ 1791	 MBDesktop *mb; 1792	 1793	 mb = mbdesktop_init(argc, argv); /media/port4 beeing where the root filesystem of the build system is mounted(could be sshfs,NSF etc...)
 * 1) 0 0x403e1254 in strlen  from /mnt/target/lib/libc.so.6
 * 2) 1 0x4003350c in mb_pixbuf_img_new_from_file (pb=0x1fd58, filename=0x0) at mbpixbuf.c:1352
 * 3) 2 0x0000c8fc in mbdesktop_set_scroll_buttons (mb=0x1b008) at mbdesktop.c:689
 * 4) 3 0x0000d334 in mbdesktop_init (argc=, argv= ) at mbdesktop.c:1022
 * 5) 4 0x0000e024 in main (argc=0, argv=0x0) at mbdesktop.c:1793
 * 1) 0 0x403e1254 in strlen  from /mnt/target/lib/libc.so.6
 * 1) 0 0x403e1254 in strlen  from /mnt/target/lib/libc.so.6
 * 2) 1 0x4003350c in mb_pixbuf_img_new_from_file (pb=0x1fd58, filename=0x0) at mbpixbuf.c:1352
 * 3) 2 0x0000c8fc in mbdesktop_set_scroll_buttons (mb=0x1b008) at mbdesktop.c:689
 * 4) 3 0x0000d334 in mbdesktop_init (argc=, argv= ) at mbdesktop.c:1022
 * 5) 4 0x0000e024 in main (argc=0, argv=0x0) at mbdesktop.c:1793
 * 1) 0 0x403e1254 in strlen  from /mnt/target/lib/libc.so.6

Howto
If you have networking,it's advised to choose it because it will be faster than serial,but if you only have serial choose serial

Core dump
Sometimes you can get a coredump. A man page describing the core dump,and in which condition it can occur can be found here In order to make possible core dumps(that unlimit the size limit on coredumps) target/device$ ulimit -c unlimited Then run your binary: target/device$ ./binary Then reproduce the conditions under which the coredump happen.The core file will appear in your current working directory and will be named core

Then copy the coredump to your workstation,and run gdb on your workstation(core is assumed to be in the current directory) ./arm-angstrom-linux-gnueabi-gdb (gdb) set sysroot /mnt/target (gdb) file /mnt/moko/usr/bin/binary (gdb) target core core

Networking
Fist start gdbserver on the target: target/device$ gdbserver :2345 ./binary arg1 arg2 Note that will expose a gdb session to all networking interfaces,for instance if you have wifi on,someone could connect to it. So if you don't want that,make it listen only on your the ip of the target interface you want to use for debugging: target/device$ gdbserver 192.168.0.202:2345 ./binary arg1 arg2 Then start GDB on your workstation,but do not connect to the target yet(else you'll have some undefined symbols): host/pc$ ./arm-angstrom-linux-gnueabi-gdb GNU gdb (Sourcery G++ Lite 2010q1-202) 7.0.50.20100218-cvs Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later  This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=i686-pc-linux-gnu --target=arm-none-linux-gnueabi". For bug reporting instructions, please see: . (gdb)

Then you need access to the target filesystem,knowing that we have networking we could use the following options: In order to use sshfs you need sftp-server installed on the target,and sshfs installed on the host computer. To mount the filesystem do: mkdir /mnt/target/ sudo sshfs root@192.168.0.202:/ /mnt/target/ -o allow_other
 * sshfs:
 * NFS(maybe you already had nfsroot on the target)
 * any other system that export the filesystem,and that is mountable

Then you need to set a variable called sysroot in order to make the cross gdb find the libraries to load (gdb) set sysroot /mnt/target Then load the binary: (gdb) file /mnt/target/usr/bin/binary Reading symbols from /mnt/target/usr/bin/binary...Reading symbols from /mnt/moko/usr/bin/.debug/binary...done. And finally connect to the target: (gdb) target remote 192.168.0.202:2345

Serial
First set the serial port speed on the target(before you issue the gdbserver command): stty speed 115200 < /dev/ttyS1 Then start gdbserver on the target: target/device$ gdbserver /dev/ttyS1 ./binary arg1 arg2 That will freeze the program until you connect with the cross gdb version and that you type "c" for continuing the execution of the program. Note that GDB will slow the program a lot. if you have a program that uses a lot of resources,it's advised to attach to it just before the crash instead like that: target/device$ gdbserver /dev/ttyS1 --attach PID Then do that on your workstation host/pc$ ./arm-angstrom-linux-gnueabi-gdb (gdb) target remote /dev/ttyS1 If you are using the serial method you need to make sure you have setup the serial speed correctly, On the host: (gdb) set remotebaud 115200

Howto
To start a new application for debug, use:

$ gdb ./binary $ gdb ./binary core.dump $ gdb --args ./binary arg1 arg2 $ gdb --command=my-gdb-commands.txt --args ./binary arg1 arg2

and then run it with (args just required if no --args were used):

(gdb) run arg1 arg2

If application is already running, find out its pid (ps, top, pidof, ...) and:

$ gdb --pid $PID

Tips
if you need to execute a series of commands every time, consider writing them on a file and use  (or  ).

Breakpoints
If you control-C (^C), it will break at that point, but you can also schedule a breakpoint with:

(gdb) break function (gdb) break line (gdb) break file:function (gdb) break file:line

conditional breaks are in the form:

(gdb) break where if condition

where  is some C expression that evaluates to 1 or 0, like

One can disable or remove breakpoints with: (gdb) enable breakpoint-number (gdb) disable breakpoint-number (gdb) delete breakpoint-number (gdb) clear      # removes all breakpoints

Examining
To list source code nearby position or specific places: (gdb) list (gdb) list line (gdb) list function (gdb) list file:line (gdb) list file:function (gdb) list *address

To list execution backtrace (or  for short): (gdb) backtrace

To change frame to operate on: (gdb) frame frame-number

To change thread to operate on: (gdb) thread thread-number

To print some value or expression: (gdb) print $register (gdb) print variable (gdb) print *address (gdb) print *(int *)address (gdb) print *(char **)address (gdb) print myfunc(p1, p2) # will actually execute it and return result! (gdb) print *a = 123 # will actually change *a value and return 123! (gdb) print file::variable (gdb) print function::variable

To disassembly: (gdb) disassembly (gdb) disassembly file:line

Print function arguments: (gdb) info args

Print locals: (gdb) info locals

Print breakpoints: (gdb) info breakpoints

Print threads: (gdb) info threads

Stepping
To go to next instruction, possible entering a function (or  for short): (gdb) step

To go to next instruction, but avoid entering new functions (or  for short): (gdb) next

To continue until the function returns: (gdb) finish

To continue execution (or  for short): (gdb) continue

Manipulating Program
To set variable to some value: (gdb) set var name=value

To force function to return: (gdb) return value (gdb) return expression

Changing Signal Handlers
(gdb) handle signal action

debugging applications with old libC, those pre-nptl, can be really annoying due SIG32 and SIG33, one can ignore those with: (gdb) handle SIG32 nostop (gdb) handle SIG32 noprint (gdb) handle SIG33 nostop (gdb) handle SIG33 noprint

Shared Object Paths
Often your cross compile root is not, so you might have to add new paths to the search list.

Unset absolute prefix: (gdb) set solib-absolute-prefix null

Add paths to search paths: (gdb) set solib-search-path /path1:/path2

Alternatively you can choose to set the prefix to the root of your target file system. Specially if you are doing embedded development and already exporting your root file system from you host machine to your target machine it can be very rewarding so simply use that as root: (gdb) set solib-absolute-prefix /rootfs

Starting application debug from host
Gdbserver could be run in multi-process mode where applications are started or processes connected to remotely, on the fly. gdbserver --multi :2345

Run crosscompiled gdb on host (/bin/arm-none-linux-gnueabi-gdb Issue the following commands (multiprocess example): file target extended remote :2345 set remote exec-file  set solib-absolute-prefix   set solib-search-path  /lib: /usr/lib break main continue