2023-04-17 15:54:12 +00:00
|
|
|
---
|
|
|
|
|
date: 2021-03-06T00:00:00-05:00
|
|
|
|
|
title: "A debuginfod service for Debian"
|
2023-04-18 16:16:48 +00:00
|
|
|
tags: [en_us, english, debian, free-software, debuginfod, gdb]
|
2023-04-17 15:54:12 +00:00
|
|
|
---
|
|
|
|
|
|
|
|
|
|
Hi there. Long time no write!
|
|
|
|
|
|
|
|
|
|
This last Tuesday, February 23, 2021, I made an [announcement at
|
|
|
|
|
debian-devel-announce](https://lists.debian.org/debian-devel-announce/2021/02/msg00003.html)
|
|
|
|
|
about a new service that I configured for Debian: [a debuginfod
|
|
|
|
|
server](https://debuginfod.debian.net).
|
|
|
|
|
|
|
|
|
|
This post serves two purposed: pay the promise I made to [Jonathan
|
|
|
|
|
Carter](https://jonathancarter.org/) that I would write a blog post
|
|
|
|
|
about the service, and go into a bit more detail about it.
|
|
|
|
|
|
|
|
|
|
What's debuginfod?
|
|
|
|
|
------------------
|
|
|
|
|
|
|
|
|
|
From the announcement above:
|
|
|
|
|
|
|
|
|
|
debuginfod is a new-ish project whose purpose is to serve
|
|
|
|
|
ELF/DWARF/source-code information over HTTP. It is developed under the
|
|
|
|
|
elfutils umbrella. You can find more information about it here:
|
|
|
|
|
|
|
|
|
|
https://sourceware.org/elfutils/Debuginfod.html
|
|
|
|
|
|
|
|
|
|
In a nutshell, by using a debuginfod service you will not need to
|
|
|
|
|
install debuginfo (a.k.a. dbgsym) files anymore; the symbols will be
|
|
|
|
|
served to GDB (or any other debuginfo consumer that supports debuginfod)
|
|
|
|
|
over the network. Ultimately, this makes the debugging experience much
|
|
|
|
|
smoother (I myself never remember the full URL of our debuginfo
|
|
|
|
|
repository when I need it).
|
|
|
|
|
|
|
|
|
|
Perhaps not everybody knows this, but until last year I was a Debugger
|
|
|
|
|
Engineer (a.k.a. *GDB hacker*) at Red Hat. I was not involved with
|
|
|
|
|
the creation of `debuginfod` directly, but I witnessed discussions
|
|
|
|
|
about "*having way to serve debug symbols over the internet*" multiple
|
|
|
|
|
times during my tenure at the company. So this is not a new idea, and
|
|
|
|
|
it's not even the first implementation, but it's the first time that
|
|
|
|
|
some engineers actually got their hands dirty enough to have something
|
|
|
|
|
concrete in hands.
|
|
|
|
|
|
|
|
|
|
The idea to set up a `debuginfod` server for Debian started to brew
|
|
|
|
|
after 2019's [GNU Tools
|
|
|
|
|
Cauldron](https://gcc.gnu.org/wiki/cauldron2019), but as usual several
|
|
|
|
|
things happened in $LIFE (including a global pandemic and leaving Red
|
|
|
|
|
Hat and starting a completely different job at Canonical) which had
|
|
|
|
|
the effect of shuffling my TODO list "a little".
|
|
|
|
|
|
|
|
|
|
Benefits for Debian
|
|
|
|
|
-------------------
|
|
|
|
|
|
|
|
|
|
Debian unfortunately is lagging behind when it comes to offer its
|
|
|
|
|
users a good debugging experience. Before the advent of our
|
|
|
|
|
`debuginfod` server, if you wanted to debug a package in Debian you
|
|
|
|
|
would need to:
|
|
|
|
|
|
|
|
|
|
1. Add the `debian-debug` apt repository to your
|
|
|
|
|
`/etc/apt/sources.list`.
|
|
|
|
|
|
|
|
|
|
2. Install the `dbgsym` package that contains the debug symbols for
|
|
|
|
|
the package you are debugging. Note that the version of the
|
|
|
|
|
`dbgsym` package needs to be **exactly** the same as the version of
|
|
|
|
|
the package you want to debug.
|
|
|
|
|
|
|
|
|
|
3. Figure out which shared libraries your package uses and install the
|
|
|
|
|
`dbgsym` packages for all of them. Arguably, this step is optional
|
|
|
|
|
but recommended if you would like to perform a more in-depth
|
|
|
|
|
debugging.
|
|
|
|
|
|
|
|
|
|
4. Download the package source, possibly using `apt source` or some
|
|
|
|
|
equivalent command.
|
|
|
|
|
|
|
|
|
|
5. Open GDB, **and make sure you adjust the source paths properly**
|
|
|
|
|
(more below). This can be non-trivial.
|
|
|
|
|
|
|
|
|
|
6. Finally, debug the program.
|
|
|
|
|
|
|
|
|
|
Now, with the new service, you will be able to start from step **4**,
|
|
|
|
|
without having to mess with `sources.list`, `dbgsym` packages and
|
|
|
|
|
version mismatches.
|
|
|
|
|
|
|
|
|
|
The package source
|
|
|
|
|
------------------
|
|
|
|
|
|
|
|
|
|
It is important to mention an existing (but perhaps not well-known)
|
|
|
|
|
limitation of our debugging experience in Debian: the need to manually
|
|
|
|
|
download the source packages **and** adjust GDB to properly find them
|
|
|
|
|
(see step **4** above). `debuginfod` is able to serve source code as
|
|
|
|
|
well, but our Debian instance is not doing that at the moment.
|
|
|
|
|
|
|
|
|
|
Debian does not provide a patched source tree that is ready to be
|
|
|
|
|
consumed by GDB nor `debuginfod` (for a good example of a distribution
|
|
|
|
|
that does that, see Fedora's `debugsource` packages). Let me show you
|
|
|
|
|
an example of debugging GDB itself (using `debuginfod`) on Debian:
|
|
|
|
|
|
2024-02-25 21:07:05 +00:00
|
|
|
```console
|
2023-04-17 15:54:12 +00:00
|
|
|
$ HOME=/tmp DEBUGINFOD_URLS=https://debuginfod.debian.net gdb -q gdb
|
|
|
|
|
Reading symbols from gdb...
|
|
|
|
|
Downloading separate debug info for /tmp/gdb...
|
|
|
|
|
Reading symbols from /tmp/.cache/debuginfod_client/02046bac4352940d19d9164bab73b2f5cefc8c73/debuginfo...
|
|
|
|
|
(gdb) start
|
|
|
|
|
Temporary breakpoint 1 at 0xd18e0: file /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c, line 28.
|
|
|
|
|
Starting program: /usr/bin/gdb
|
|
|
|
|
Downloading separate debug info for /lib/x86_64-linux-gnu/libreadline.so.8...
|
|
|
|
|
Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1...
|
|
|
|
|
Downloading separate debug info for /lib/x86_64-linux-gnu/libncursesw.so.6...
|
|
|
|
|
Downloading separate debug info for /lib/x86_64-linux-gnu/libtinfo.so.6...
|
|
|
|
|
Downloading separate debug info for /tmp/.cache/debuginfod_client/d6920dbdd057f44edaf4c1fbce191b5854dfd9e6/debuginfo...
|
|
|
|
|
[Thread debugging using libthread_db enabled]
|
|
|
|
|
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
|
|
|
|
|
Downloading separate debug info for /lib/x86_64-linux-gnu/libexpat.so.1...
|
|
|
|
|
Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbabeltrace.so.1...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbabeltrace-ctf.so.1...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libipt.so.2...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libmpfr.so.6...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libsource-highlight.so.4...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libxxhash.so.0...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libdebuginfod.so.1...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libstdc++.so.6...
|
|
|
|
|
Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0...
|
|
|
|
|
Downloading separate debug info for /tmp/.cache/debuginfod_client/dbfea245d26065975b4084f4e9cd2d83c65973ee/debuginfo...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libdw.so.1...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libelf.so.1...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libuuid.so.1...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libgmp.so.10...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libboost_regex.so.1.74.0...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4...
|
|
|
|
|
Downloading separate debug info for /lib/x86_64-linux-gnu/libbz2.so.1.0...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libicui18n.so.67...
|
|
|
|
|
Downloading separate debug info for /tmp/.cache/debuginfod_client/acaa831dbbc8aa70bb2131134e0c83206a0701f9/debuginfo...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libicuuc.so.67...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libnghttp2.so.14...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libidn2.so.0...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/librtmp.so.1...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libssh2.so.1...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libpsl.so.5...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libnettle.so.8...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libgnutls.so.30...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbrotlidec.so.1...
|
|
|
|
|
Downloading separate debug info for /tmp/.cache/debuginfod_client/39739740c2f8a033de95c1c0b1eb8be445610b31/debuginfo...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libunistring.so.2...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libhogweed.so.6...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libgcrypt.so.20...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libp11-kit.so.0...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libtasn1.so.6...
|
|
|
|
|
Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libsasl2.so.2...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbrotlicommon.so.1...
|
|
|
|
|
Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0...
|
|
|
|
|
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libffi.so.7...
|
|
|
|
|
Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1...
|
|
|
|
|
|
|
|
|
|
Temporary breakpoint 1, main (argc=1, argv=0x7fffffffebf8) at /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c:28
|
|
|
|
|
28 /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c: Directory not empty.
|
|
|
|
|
(gdb) list
|
|
|
|
|
23 in /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c
|
|
|
|
|
(gdb)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
(See all those `Downloading separate debug info for...` lines? Nice!)
|
|
|
|
|
|
|
|
|
|
As you can see, when we try to `list` the contents of the file we're
|
|
|
|
|
in, nothing shows up. This happens because GDB doesn't know where the
|
|
|
|
|
file is. So you have to tell it. In this case, it's relatively easy:
|
|
|
|
|
you see that the GDB package's build directory is
|
|
|
|
|
`/build/gdb-Nav6Es/gdb-10.1/`. When you `apt source gdb`, you will
|
|
|
|
|
have a directory called `$PWD/gdb-10.1/` containing the full source of the
|
|
|
|
|
package. Notice that the last directory's name in both paths is the
|
|
|
|
|
same, so in this case we can use GDB's `set substitute-path` command
|
|
|
|
|
do the job for us (in this example `$PWD` is `/tmp/`):
|
|
|
|
|
|
2024-02-25 21:07:05 +00:00
|
|
|
```console
|
2023-04-17 15:54:12 +00:00
|
|
|
$ HOME=/tmp DEBUGINFOD_URLS=https://debuginfod.debian.net gdb -q gdb
|
|
|
|
|
Reading symbols from gdb...
|
|
|
|
|
Reading symbols from /tmp/.cache/debuginfod_client/02046bac4352940d19d9164bab73b2f5cefc8c73/debuginfo...
|
|
|
|
|
(gdb) set substitute-path /build/gdb-Nav6Es/ /tmp/
|
|
|
|
|
(gdb) start
|
|
|
|
|
Temporary breakpoint 1 at 0xd18e0: file /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c, line 28.
|
|
|
|
|
Starting program: /usr/bin/gdb
|
|
|
|
|
[Thread debugging using libthread_db enabled]
|
|
|
|
|
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
|
|
|
|
|
|
|
|
|
|
Temporary breakpoint 1, main (argc=1, argv=0x7fffffffebf8) at /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c:28
|
|
|
|
|
warning: Source file is more recent than executable.
|
|
|
|
|
28 memset (&args, 0, sizeof args);
|
|
|
|
|
(gdb) list
|
|
|
|
|
23 int
|
|
|
|
|
24 main (int argc, char **argv)
|
|
|
|
|
25 {
|
|
|
|
|
26 struct captured_main_args args;
|
|
|
|
|
27
|
|
|
|
|
28 memset (&args, 0, sizeof args);
|
|
|
|
|
29 args.argc = argc;
|
|
|
|
|
30 args.argv = argv;
|
|
|
|
|
31 args.interpreter_p = INTERP_CONSOLE;
|
|
|
|
|
32 return gdb_main (&args);
|
|
|
|
|
(gdb)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Much better, huh? The problem is that this process is manual, and
|
|
|
|
|
will change depending on how the package you're debugging was built.
|
|
|
|
|
|
|
|
|
|
What can we do to improve this? What I personally would like to see
|
|
|
|
|
is something similar to what the Fedora project already does: create a
|
|
|
|
|
new debug package which will contain the full, patched source package.
|
|
|
|
|
This would mean changing our building infrastructure and possibly
|
|
|
|
|
other somewhat complex things.
|
|
|
|
|
|
|
|
|
|
Using the service (by default)
|
|
|
|
|
-----------------------------
|
|
|
|
|
|
|
|
|
|
At the time of this writing, I am working on an `elfutils` [Merge
|
|
|
|
|
Request](https://salsa.debian.org/toolchain-team/elfutils/-/merge_requests/2)
|
|
|
|
|
whose purpose is to implement a debconf question to ask the user
|
|
|
|
|
whether she wants to use our service by default.
|
|
|
|
|
|
|
|
|
|
If you would like to start using the service right now, all you have
|
|
|
|
|
to do is set the following environment variable in your shell:
|
|
|
|
|
|
2024-02-25 21:07:05 +00:00
|
|
|
```bash
|
2023-04-17 15:54:12 +00:00
|
|
|
DEBUGINFOD_URLS="https://debuginfod.debian.net"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
More information
|
|
|
|
|
----------------
|
|
|
|
|
|
|
|
|
|
You can find more information about our `debuginfod` service
|
|
|
|
|
[here](https://wiki.debian.org/Debuginfod). Try to keep an eye on the
|
|
|
|
|
page as it's being constantly updated.
|
|
|
|
|
|
|
|
|
|
If you'd like to get in touch with me, my email is my domain at debian
|
|
|
|
|
dot org.
|
|
|
|
|
|
|
|
|
|
I sincerely believe that this service is a step in the right
|
|
|
|
|
direction, and hope that it can be useful to you :-).
|
