2023-04-17 15:54:12 +00:00
---
date: 2019-05-06T00:00:00-05:00
title: "Improve gcore and support dumping ELF headers"
2023-04-18 16:16:48 +00:00
tags: [debian, fedora-planet, free-software, gdb, linux, en_us, english]
2023-04-17 15:54:12 +00:00
---
Back in 2016, when life was simpler, a Fedora GDB user
reported [a bug ](https://bugzilla.redhat.com/show_bug.cgi?id=1371380 )
(or a feature request, depending on how you interpret it) saying that
GDB's `gcore` command did not respect the `COREFILTER_ELF_HEADERS`
flag, which instructs it to dump memory pages containing ELF headers.
As you may or may not remember, I have
already
[written about the broader topic of revamping GDB's internal corefile dump algorithm ]({filename}/2015-04-05-linux-memory-mapping.md );
it's an interesting read and I recommend it if you don't know how
Linux (or GDB) decides which mappings to dump to a corefile.
Anyway, even though the bug was interesting and had to do with a work
I'd done before, I couldn't really work on it at the time, so I
decided to put it in the TODO list. Of course, the "TODO list" is
actually a crack where most things fall through and are usually never
seen again, so I was blissfully ignoring this request because I had
other major priorities to deal with. That is, until a seemingly
unrelated problem forced me to face this once and for all!
What? A regression? Since when?
---------------------------------
As the Fedora GDB maintainer, I'm routinely preparing new releases for
Fedora Rawhide distribution, and sometimes for the stable versions of
the distro as well. And I try to be very careful when dealing with
new releases, because a regression introduced now can come and bite us
(i.e., the Red Hat GDB team) back many years in the future, when it's
sometimes too late or too difficult to fix things. So, a mandatory
part of every release preparation is to actually run a regression test
against the previous release, and make sure that everything is working
correctly.
One of these days, some weeks ago, I had finished running the
regression check for the release I was preparing when I noticed
something strange: a specific, Fedora-only corefile test was FAILing.
That's a no-no, so I started investigating and found that the
underlying reason was that, when the corefile was being generated,
the [build-id ](https://fedoraproject.org/wiki/Releases/FeatureBuildId )
note from the executable was not being copied over. Fedora GDB has a
local patch whose job is to, given a corefile with a build-id note,
locate the corresponding binary that generated it. Without the
build-id note, no binary was being located.
Coincidentally or not, at the same I started noticing some users
reporting very similar build-id issues on the freenode's `#gdb`
channel, and I thought that this bug had a potential to become a big
headache for us if nothing was done to fix it right now.
I asked for some help from the team, and we managed to discover that
the problem was also happening with upstream `gcore` , and that it was
probably something that **binutils** was doing, and not GDB. Hmm...
Ah, so it's `ld` 's fault. Or is it?
------------------------------------
So there I went, trying to confirm that it was binutils's fault, and
not GDB's. Of course, if I could confirm this, then I could also tell
the binutils guys to fix it, which meant less work for us :-).
With a lot of help from Keith Seitz, I was able to bisect the problem
and found that it started with the following commit:
2024-02-25 21:07:05 +00:00
```
commit f6aec96dce1ddbd8961a3aa8a2925db2021719bb
Author: H.J. Lu < hjl.tools @ gmail . com >
Date: Tue Feb 27 11:34:20 2018 -0800
2023-04-17 15:54:12 +00:00
2024-02-25 21:07:05 +00:00
ld: Add --enable-separate-code
```
2023-04-17 15:54:12 +00:00
This is a commit that touches the linker, which is part of binutils.
So that means this is not GDB's problem, right?!? Hmm. No,
unfortunately not.
What the commit above does is to simply enable the use of
`--enable-separate-code` (or `-z separate-code` ) by default when
linking an ELF program on x86_64 (more on that later). On a first
glance, this change should not impact the corefile generation, and
indeed, if you tell the Linux kernel to generate a corefile (for
example, by doing `sleep 60 &` and then hitting `C-\` ), you will
notice that the build-id note **is** included into it! So GDB was
still a suspect here. The investigation needed to continue.
What's with `-z separate-code` ?
-------------------------------
The `-z separate-code` option makes the code segment in the ELF file
to put in a completely separated segment than data segment. This was
done to increase the security of generated binaries. Before it,
everything (code and data) was put together in the same memory
region. What this means in practice is that, before, you would see
something like this when you examined `/proc/PID/smaps` :
2024-02-25 21:07:05 +00:00
```
00400000-00401000 r-xp 00000000 fc:01 798593 /file
Size: 4 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Rss: 4 kB
Pss: 4 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 4 kB
Referenced: 4 kB
Anonymous: 4 kB
LazyFree: 0 kB
AnonHugePages: 0 kB
ShmemPmdMapped: 0 kB
Shared_Hugetlb: 0 kB
Private_Hugetlb: 0 kB
Swap: 0 kB
SwapPss: 0 kB
Locked: 0 kB
THPeligible: 0
VmFlags: rd ex mr mw me dw sd
```
2023-04-17 15:54:12 +00:00
And now, you will see two memory regions instead, like this:
2024-02-25 21:07:05 +00:00
```
00400000-00401000 r--p 00000000 fc:01 799548 /file
Size: 4 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Rss: 4 kB
Pss: 4 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 4 kB
Private_Dirty: 0 kB
Referenced: 4 kB
Anonymous: 0 kB
LazyFree: 0 kB
AnonHugePages: 0 kB
ShmemPmdMapped: 0 kB
Shared_Hugetlb: 0 kB
Private_Hugetlb: 0 kB
Swap: 0 kB
SwapPss: 0 kB
Locked: 0 kB
THPeligible: 0
VmFlags: rd mr mw me dw sd
00401000-00402000 r-xp 00001000 fc:01 799548 /file
Size: 4 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Rss: 4 kB
Pss: 4 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 4 kB
Referenced: 4 kB
Anonymous: 4 kB
LazyFree: 0 kB
AnonHugePages: 0 kB
ShmemPmdMapped: 0 kB
Shared_Hugetlb: 0 kB
Private_Hugetlb: 0 kB
Swap: 0 kB
SwapPss: 0 kB
Locked: 0 kB
THPeligible: 0
VmFlags: rd ex mr mw me dw sd
```
2023-04-17 15:54:12 +00:00
A few minor things have changed, but the most important of them is the
fact that, before, the whole memory region **had** anonymous data in
it, which means that it was considered an **anonymous private
mapping** (**anonymous** because of the non-zero Anonymous amount of
data; **private** because of the `p` in the `r-xp` permission bits).
After `-z separate-code` was made default, the first memory mapping
does **not** have Anonymous contents anymore, which means that it is
now considered to be a **file-backed private** mapping instead.
GDB, corefile, and coredump_filter
----------------------------------
It is important to mention that, unlike the Linux kernel, GDB doesn't
have all of the necessary information readily available to decide the
exact type of a memory mapping, so when I revamped this code back in
2015 I had to create some heuristics to try and determine this
information. If you're curious, take a look at the `linux-tdep.c`
file on GDB's source tree, specifically at the
functions
[`dump_mapping_p` ](https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/linux-tdep.c;h=c1666d189ae009b594d906ca7a87091ea535e05f;hb=HEAD#l588 ) and
[`linux_find_memory_regions_full` ](https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/linux-tdep.c;h=c1666d189ae009b594d906ca7a87091ea535e05f;hb=HEAD#l1200 ).
When GDB is deciding which memory regions should be dumped into the
corefile, it respects the value found at the
`/proc/PID/coredump_filter` file. The default value for this file is
`0x33` , which, according to `core(5)` , means:
Dump memory pages that are either anonymous private, anonymous
shared, ELF headers or HugeTLB.
GDB had the support implemented to dump almost all of these pages,
except for the ELF headers variety. And, as you can probably infer,
this means that, before the `-z separate-code` change, the very first
memory mapping of the executable **was** being dumped, because it was
marked as anonymous private. However, after the change, the first
mapping (which contains only data, no code) wasn't being dumped
anymore, because it was now considered by GDB to be a file-backed
private mapping!
Finally, that is the reason for the difference between corefiles
generated by GDB and Linux, and also the reason why the build-id note
was not being included in the corefile anymore! You see, the first
memory mapping contains not only the program's data, but also its ELF
headers, which in turn contain the build-id information.
`gcore` , meet ELF headers
-------------------------
The solution was "simple": I needed to improve the current heuristics
and teach GDB how to determine if a mapping contains an ELF header or
not. For that, I chose to follow the Linux kernel's algorithm, which
basically checks the first 4 bytes of the mapping and compares them
against `\177ELF` , which is ELF's magic number. If the comparison
succeeds, then we just assume we're dealing with a mapping that
contains an ELF header and dump it.
In all fairness, Linux just dumps the first page (4K) of the mapping,
in order to save space. It would be possible to make GDB do the same,
but I chose the faster way and just dumped the whole mapping, which,
in most scenarios, shouldn't be a big problem.
It's also interesting to mention that GDB will just perform this check
if:
* The heuristic has decided *not* to dump the mapping so far, and;
* The mapping is private, and;
* The mapping's offset is zero, and;
* There is a request to dump mappings with ELF headers (i.e.,
`coredump_filter` ).
Linux also makes these checks, by the way.
The patch, finally
------------------
2024-02-25 21:07:05 +00:00
I submitted [the
patch](https://sourceware.org/ml/gdb-patches/2019-04/msg00479.html) to
2023-04-17 15:54:12 +00:00
the mailing list, and it was approved fairly quickly (with a few minor
nits).
The reason I'm writing this blog post is because I'm very happy and
proud with the whole process. It wasn't an easy task to investigate
the underlying reason for the build-id failures, and it was
interesting to come up with a solution that extended the work I did a
few years ago. I was also able to close a few bug reports upstream,
as well as the one reported against Fedora GDB.
2024-02-25 21:07:05 +00:00
The patch has been
2023-04-17 15:54:12 +00:00
[pushed ](https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=57e5e645010430b3d73f8c6a757d09f48dc8f8d5 ),
and is also present at the latest version of Fedora GDB for Rawhide.
It wasn't possible to write a self-contained testcase for this
problem, so I had to resort to using an external tool (`eu-unstrip`)
in order to guarantee that the build-id note is correctly present in
the corefile. But that's a small detail, of course.
Anyway, I hope this was an interesting (albeit large) read!
