--- date: 2021-03-06T00:00:00-05:00 title: "A debuginfod service for Debian" tags: [en_us, english, debian, free-software, debuginfod, gdb] --- Hi there. Long time no write! This last Tuesday, February 23, 2021, I made an [announcement at debian-devel-announce](https://lists.debian.org/debian-devel-announce/2021/02/msg00003.html) about a new service that I configured for Debian: [a debuginfod server](https://debuginfod.debian.net). This post serves two purposed: pay the promise I made to [Jonathan Carter](https://jonathancarter.org/) that I would write a blog post about the service, and go into a bit more detail about it. What's debuginfod? ------------------ From the announcement above: debuginfod is a new-ish project whose purpose is to serve ELF/DWARF/source-code information over HTTP. It is developed under the elfutils umbrella. You can find more information about it here: https://sourceware.org/elfutils/Debuginfod.html In a nutshell, by using a debuginfod service you will not need to install debuginfo (a.k.a. dbgsym) files anymore; the symbols will be served to GDB (or any other debuginfo consumer that supports debuginfod) over the network. Ultimately, this makes the debugging experience much smoother (I myself never remember the full URL of our debuginfo repository when I need it). Perhaps not everybody knows this, but until last year I was a Debugger Engineer (a.k.a. *GDB hacker*) at Red Hat. I was not involved with the creation of `debuginfod` directly, but I witnessed discussions about "*having way to serve debug symbols over the internet*" multiple times during my tenure at the company. So this is not a new idea, and it's not even the first implementation, but it's the first time that some engineers actually got their hands dirty enough to have something concrete in hands. The idea to set up a `debuginfod` server for Debian started to brew after 2019's [GNU Tools Cauldron](https://gcc.gnu.org/wiki/cauldron2019), but as usual several things happened in $LIFE (including a global pandemic and leaving Red Hat and starting a completely different job at Canonical) which had the effect of shuffling my TODO list "a little". Benefits for Debian ------------------- Debian unfortunately is lagging behind when it comes to offer its users a good debugging experience. Before the advent of our `debuginfod` server, if you wanted to debug a package in Debian you would need to: 1. Add the `debian-debug` apt repository to your `/etc/apt/sources.list`. 2. Install the `dbgsym` package that contains the debug symbols for the package you are debugging. Note that the version of the `dbgsym` package needs to be **exactly** the same as the version of the package you want to debug. 3. Figure out which shared libraries your package uses and install the `dbgsym` packages for all of them. Arguably, this step is optional but recommended if you would like to perform a more in-depth debugging. 4. Download the package source, possibly using `apt source` or some equivalent command. 5. Open GDB, **and make sure you adjust the source paths properly** (more below). This can be non-trivial. 6. Finally, debug the program. Now, with the new service, you will be able to start from step **4**, without having to mess with `sources.list`, `dbgsym` packages and version mismatches. The package source ------------------ It is important to mention an existing (but perhaps not well-known) limitation of our debugging experience in Debian: the need to manually download the source packages **and** adjust GDB to properly find them (see step **4** above). `debuginfod` is able to serve source code as well, but our Debian instance is not doing that at the moment. Debian does not provide a patched source tree that is ready to be consumed by GDB nor `debuginfod` (for a good example of a distribution that does that, see Fedora's `debugsource` packages). Let me show you an example of debugging GDB itself (using `debuginfod`) on Debian: ```console $ HOME=/tmp DEBUGINFOD_URLS=https://debuginfod.debian.net gdb -q gdb Reading symbols from gdb... Downloading separate debug info for /tmp/gdb... Reading symbols from /tmp/.cache/debuginfod_client/02046bac4352940d19d9164bab73b2f5cefc8c73/debuginfo... (gdb) start Temporary breakpoint 1 at 0xd18e0: file /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c, line 28. Starting program: /usr/bin/gdb Downloading separate debug info for /lib/x86_64-linux-gnu/libreadline.so.8... Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1... Downloading separate debug info for /lib/x86_64-linux-gnu/libncursesw.so.6... Downloading separate debug info for /lib/x86_64-linux-gnu/libtinfo.so.6... Downloading separate debug info for /tmp/.cache/debuginfod_client/d6920dbdd057f44edaf4c1fbce191b5854dfd9e6/debuginfo... [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Downloading separate debug info for /lib/x86_64-linux-gnu/libexpat.so.1... Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbabeltrace.so.1... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbabeltrace-ctf.so.1... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libipt.so.2... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libmpfr.so.6... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libsource-highlight.so.4... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libxxhash.so.0... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libdebuginfod.so.1... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libstdc++.so.6... Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0... Downloading separate debug info for /tmp/.cache/debuginfod_client/dbfea245d26065975b4084f4e9cd2d83c65973ee/debuginfo... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libdw.so.1... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libelf.so.1... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libuuid.so.1... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libgmp.so.10... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libboost_regex.so.1.74.0... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4... Downloading separate debug info for /lib/x86_64-linux-gnu/libbz2.so.1.0... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libicui18n.so.67... Downloading separate debug info for /tmp/.cache/debuginfod_client/acaa831dbbc8aa70bb2131134e0c83206a0701f9/debuginfo... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libicuuc.so.67... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libnghttp2.so.14... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libidn2.so.0... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/librtmp.so.1... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libssh2.so.1... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libpsl.so.5... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libnettle.so.8... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libgnutls.so.30... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbrotlidec.so.1... Downloading separate debug info for /tmp/.cache/debuginfod_client/39739740c2f8a033de95c1c0b1eb8be445610b31/debuginfo... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libunistring.so.2... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libhogweed.so.6... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libgcrypt.so.20... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libp11-kit.so.0... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libtasn1.so.6... Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libsasl2.so.2... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbrotlicommon.so.1... Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0... Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libffi.so.7... Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1... Temporary breakpoint 1, main (argc=1, argv=0x7fffffffebf8) at /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c:28 28 /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c: Directory not empty. (gdb) list 23 in /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c (gdb) ``` (See all those `Downloading separate debug info for...` lines? Nice!) As you can see, when we try to `list` the contents of the file we're in, nothing shows up. This happens because GDB doesn't know where the file is. So you have to tell it. In this case, it's relatively easy: you see that the GDB package's build directory is `/build/gdb-Nav6Es/gdb-10.1/`. When you `apt source gdb`, you will have a directory called `$PWD/gdb-10.1/` containing the full source of the package. Notice that the last directory's name in both paths is the same, so in this case we can use GDB's `set substitute-path` command do the job for us (in this example `$PWD` is `/tmp/`): ```console $ HOME=/tmp DEBUGINFOD_URLS=https://debuginfod.debian.net gdb -q gdb Reading symbols from gdb... Reading symbols from /tmp/.cache/debuginfod_client/02046bac4352940d19d9164bab73b2f5cefc8c73/debuginfo... (gdb) set substitute-path /build/gdb-Nav6Es/ /tmp/ (gdb) start Temporary breakpoint 1 at 0xd18e0: file /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c, line 28. Starting program: /usr/bin/gdb [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Temporary breakpoint 1, main (argc=1, argv=0x7fffffffebf8) at /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c:28 warning: Source file is more recent than executable. 28 memset (&args, 0, sizeof args); (gdb) list 23 int 24 main (int argc, char **argv) 25 { 26 struct captured_main_args args; 27 28 memset (&args, 0, sizeof args); 29 args.argc = argc; 30 args.argv = argv; 31 args.interpreter_p = INTERP_CONSOLE; 32 return gdb_main (&args); (gdb) ``` Much better, huh? The problem is that this process is manual, and will change depending on how the package you're debugging was built. What can we do to improve this? What I personally would like to see is something similar to what the Fedora project already does: create a new debug package which will contain the full, patched source package. This would mean changing our building infrastructure and possibly other somewhat complex things. Using the service (by default) ----------------------------- At the time of this writing, I am working on an `elfutils` [Merge Request](https://salsa.debian.org/toolchain-team/elfutils/-/merge_requests/2) whose purpose is to implement a debconf question to ask the user whether she wants to use our service by default. If you would like to start using the service right now, all you have to do is set the following environment variable in your shell: ```bash DEBUGINFOD_URLS="https://debuginfod.debian.net" ``` More information ---------------- You can find more information about our `debuginfod` service [here](https://wiki.debian.org/Debuginfod). Try to keep an eye on the page as it's being constantly updated. If you'd like to get in touch with me, my email is my domain at debian dot org. I sincerely believe that this service is a step in the right direction, and hope that it can be useful to you :-).