269 lines
11 KiB
Markdown
269 lines
11 KiB
Markdown
---
|
|
date: 2019-05-06T00:00:00-05:00
|
|
title: "Improve gcore and support dumping ELF headers"
|
|
tags: [debian, fedora-planet, free-software, gdb, linux, en_us, english]
|
|
---
|
|
|
|
Back in 2016, when life was simpler, a Fedora GDB user
|
|
reported [a bug](https://bugzilla.redhat.com/show_bug.cgi?id=1371380)
|
|
(or a feature request, depending on how you interpret it) saying that
|
|
GDB's `gcore` command did not respect the `COREFILTER_ELF_HEADERS`
|
|
flag, which instructs it to dump memory pages containing ELF headers.
|
|
As you may or may not remember, I have
|
|
already
|
|
[written about the broader topic of revamping GDB's internal corefile dump algorithm]({filename}/2015-04-05-linux-memory-mapping.md);
|
|
it's an interesting read and I recommend it if you don't know how
|
|
Linux (or GDB) decides which mappings to dump to a corefile.
|
|
|
|
Anyway, even though the bug was interesting and had to do with a work
|
|
I'd done before, I couldn't really work on it at the time, so I
|
|
decided to put it in the TODO list. Of course, the "TODO list" is
|
|
actually a crack where most things fall through and are usually never
|
|
seen again, so I was blissfully ignoring this request because I had
|
|
other major priorities to deal with. That is, until a seemingly
|
|
unrelated problem forced me to face this once and for all!
|
|
|
|
What? A regression? Since when?
|
|
---------------------------------
|
|
|
|
As the Fedora GDB maintainer, I'm routinely preparing new releases for
|
|
Fedora Rawhide distribution, and sometimes for the stable versions of
|
|
the distro as well. And I try to be very careful when dealing with
|
|
new releases, because a regression introduced now can come and bite us
|
|
(i.e., the Red Hat GDB team) back many years in the future, when it's
|
|
sometimes too late or too difficult to fix things. So, a mandatory
|
|
part of every release preparation is to actually run a regression test
|
|
against the previous release, and make sure that everything is working
|
|
correctly.
|
|
|
|
One of these days, some weeks ago, I had finished running the
|
|
regression check for the release I was preparing when I noticed
|
|
something strange: a specific, Fedora-only corefile test was FAILing.
|
|
That's a no-no, so I started investigating and found that the
|
|
underlying reason was that, when the corefile was being generated,
|
|
the [build-id](https://fedoraproject.org/wiki/Releases/FeatureBuildId)
|
|
note from the executable was not being copied over. Fedora GDB has a
|
|
local patch whose job is to, given a corefile with a build-id note,
|
|
locate the corresponding binary that generated it. Without the
|
|
build-id note, no binary was being located.
|
|
|
|
Coincidentally or not, at the same I started noticing some users
|
|
reporting very similar build-id issues on the freenode's `#gdb`
|
|
channel, and I thought that this bug had a potential to become a big
|
|
headache for us if nothing was done to fix it right now.
|
|
|
|
I asked for some help from the team, and we managed to discover that
|
|
the problem was also happening with upstream `gcore`, and that it was
|
|
probably something that **binutils** was doing, and not GDB. Hmm...
|
|
|
|
Ah, so it's `ld`'s fault. Or is it?
|
|
------------------------------------
|
|
|
|
So there I went, trying to confirm that it was binutils's fault, and
|
|
not GDB's. Of course, if I could confirm this, then I could also tell
|
|
the binutils guys to fix it, which meant less work for us :-).
|
|
|
|
With a lot of help from Keith Seitz, I was able to bisect the problem
|
|
and found that it started with the following commit:
|
|
|
|
```
|
|
commit f6aec96dce1ddbd8961a3aa8a2925db2021719bb
|
|
Author: H.J. Lu <hjl.tools@gmail.com>
|
|
Date: Tue Feb 27 11:34:20 2018 -0800
|
|
|
|
ld: Add --enable-separate-code
|
|
```
|
|
|
|
This is a commit that touches the linker, which is part of binutils.
|
|
So that means this is not GDB's problem, right?!? Hmm. No,
|
|
unfortunately not.
|
|
|
|
What the commit above does is to simply enable the use of
|
|
`--enable-separate-code` (or `-z separate-code`) by default when
|
|
linking an ELF program on x86_64 (more on that later). On a first
|
|
glance, this change should not impact the corefile generation, and
|
|
indeed, if you tell the Linux kernel to generate a corefile (for
|
|
example, by doing `sleep 60 &` and then hitting `C-\`), you will
|
|
notice that the build-id note **is** included into it! So GDB was
|
|
still a suspect here. The investigation needed to continue.
|
|
|
|
What's with `-z separate-code`?
|
|
-------------------------------
|
|
|
|
The `-z separate-code` option makes the code segment in the ELF file
|
|
to put in a completely separated segment than data segment. This was
|
|
done to increase the security of generated binaries. Before it,
|
|
everything (code and data) was put together in the same memory
|
|
region. What this means in practice is that, before, you would see
|
|
something like this when you examined `/proc/PID/smaps`:
|
|
|
|
```
|
|
00400000-00401000 r-xp 00000000 fc:01 798593 /file
|
|
Size: 4 kB
|
|
KernelPageSize: 4 kB
|
|
MMUPageSize: 4 kB
|
|
Rss: 4 kB
|
|
Pss: 4 kB
|
|
Shared_Clean: 0 kB
|
|
Shared_Dirty: 0 kB
|
|
Private_Clean: 0 kB
|
|
Private_Dirty: 4 kB
|
|
Referenced: 4 kB
|
|
Anonymous: 4 kB
|
|
LazyFree: 0 kB
|
|
AnonHugePages: 0 kB
|
|
ShmemPmdMapped: 0 kB
|
|
Shared_Hugetlb: 0 kB
|
|
Private_Hugetlb: 0 kB
|
|
Swap: 0 kB
|
|
SwapPss: 0 kB
|
|
Locked: 0 kB
|
|
THPeligible: 0
|
|
VmFlags: rd ex mr mw me dw sd
|
|
```
|
|
|
|
And now, you will see two memory regions instead, like this:
|
|
|
|
```
|
|
00400000-00401000 r--p 00000000 fc:01 799548 /file
|
|
Size: 4 kB
|
|
KernelPageSize: 4 kB
|
|
MMUPageSize: 4 kB
|
|
Rss: 4 kB
|
|
Pss: 4 kB
|
|
Shared_Clean: 0 kB
|
|
Shared_Dirty: 0 kB
|
|
Private_Clean: 4 kB
|
|
Private_Dirty: 0 kB
|
|
Referenced: 4 kB
|
|
Anonymous: 0 kB
|
|
LazyFree: 0 kB
|
|
AnonHugePages: 0 kB
|
|
ShmemPmdMapped: 0 kB
|
|
Shared_Hugetlb: 0 kB
|
|
Private_Hugetlb: 0 kB
|
|
Swap: 0 kB
|
|
SwapPss: 0 kB
|
|
Locked: 0 kB
|
|
THPeligible: 0
|
|
VmFlags: rd mr mw me dw sd
|
|
00401000-00402000 r-xp 00001000 fc:01 799548 /file
|
|
Size: 4 kB
|
|
KernelPageSize: 4 kB
|
|
MMUPageSize: 4 kB
|
|
Rss: 4 kB
|
|
Pss: 4 kB
|
|
Shared_Clean: 0 kB
|
|
Shared_Dirty: 0 kB
|
|
Private_Clean: 0 kB
|
|
Private_Dirty: 4 kB
|
|
Referenced: 4 kB
|
|
Anonymous: 4 kB
|
|
LazyFree: 0 kB
|
|
AnonHugePages: 0 kB
|
|
ShmemPmdMapped: 0 kB
|
|
Shared_Hugetlb: 0 kB
|
|
Private_Hugetlb: 0 kB
|
|
Swap: 0 kB
|
|
SwapPss: 0 kB
|
|
Locked: 0 kB
|
|
THPeligible: 0
|
|
VmFlags: rd ex mr mw me dw sd
|
|
```
|
|
|
|
A few minor things have changed, but the most important of them is the
|
|
fact that, before, the whole memory region **had** anonymous data in
|
|
it, which means that it was considered an **anonymous private
|
|
mapping** (**anonymous** because of the non-zero Anonymous amount of
|
|
data; **private** because of the `p` in the `r-xp` permission bits).
|
|
After `-z separate-code` was made default, the first memory mapping
|
|
does **not** have Anonymous contents anymore, which means that it is
|
|
now considered to be a **file-backed private** mapping instead.
|
|
|
|
GDB, corefile, and coredump_filter
|
|
----------------------------------
|
|
|
|
It is important to mention that, unlike the Linux kernel, GDB doesn't
|
|
have all of the necessary information readily available to decide the
|
|
exact type of a memory mapping, so when I revamped this code back in
|
|
2015 I had to create some heuristics to try and determine this
|
|
information. If you're curious, take a look at the `linux-tdep.c`
|
|
file on GDB's source tree, specifically at the
|
|
functions
|
|
[`dump_mapping_p`](https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/linux-tdep.c;h=c1666d189ae009b594d906ca7a87091ea535e05f;hb=HEAD#l588) and
|
|
[`linux_find_memory_regions_full`](https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/linux-tdep.c;h=c1666d189ae009b594d906ca7a87091ea535e05f;hb=HEAD#l1200).
|
|
|
|
When GDB is deciding which memory regions should be dumped into the
|
|
corefile, it respects the value found at the
|
|
`/proc/PID/coredump_filter` file. The default value for this file is
|
|
`0x33`, which, according to `core(5)`, means:
|
|
|
|
Dump memory pages that are either anonymous private, anonymous
|
|
shared, ELF headers or HugeTLB.
|
|
|
|
GDB had the support implemented to dump almost all of these pages,
|
|
except for the ELF headers variety. And, as you can probably infer,
|
|
this means that, before the `-z separate-code` change, the very first
|
|
memory mapping of the executable **was** being dumped, because it was
|
|
marked as anonymous private. However, after the change, the first
|
|
mapping (which contains only data, no code) wasn't being dumped
|
|
anymore, because it was now considered by GDB to be a file-backed
|
|
private mapping!
|
|
|
|
Finally, that is the reason for the difference between corefiles
|
|
generated by GDB and Linux, and also the reason why the build-id note
|
|
was not being included in the corefile anymore! You see, the first
|
|
memory mapping contains not only the program's data, but also its ELF
|
|
headers, which in turn contain the build-id information.
|
|
|
|
`gcore`, meet ELF headers
|
|
-------------------------
|
|
|
|
The solution was "simple": I needed to improve the current heuristics
|
|
and teach GDB how to determine if a mapping contains an ELF header or
|
|
not. For that, I chose to follow the Linux kernel's algorithm, which
|
|
basically checks the first 4 bytes of the mapping and compares them
|
|
against `\177ELF`, which is ELF's magic number. If the comparison
|
|
succeeds, then we just assume we're dealing with a mapping that
|
|
contains an ELF header and dump it.
|
|
|
|
In all fairness, Linux just dumps the first page (4K) of the mapping,
|
|
in order to save space. It would be possible to make GDB do the same,
|
|
but I chose the faster way and just dumped the whole mapping, which,
|
|
in most scenarios, shouldn't be a big problem.
|
|
|
|
It's also interesting to mention that GDB will just perform this check
|
|
if:
|
|
|
|
* The heuristic has decided *not* to dump the mapping so far, and;
|
|
* The mapping is private, and;
|
|
* The mapping's offset is zero, and;
|
|
* There is a request to dump mappings with ELF headers (i.e.,
|
|
`coredump_filter`).
|
|
|
|
Linux also makes these checks, by the way.
|
|
|
|
The patch, finally
|
|
------------------
|
|
|
|
I submitted [the
|
|
patch](https://sourceware.org/ml/gdb-patches/2019-04/msg00479.html) to
|
|
the mailing list, and it was approved fairly quickly (with a few minor
|
|
nits).
|
|
|
|
The reason I'm writing this blog post is because I'm very happy and
|
|
proud with the whole process. It wasn't an easy task to investigate
|
|
the underlying reason for the build-id failures, and it was
|
|
interesting to come up with a solution that extended the work I did a
|
|
few years ago. I was also able to close a few bug reports upstream,
|
|
as well as the one reported against Fedora GDB.
|
|
|
|
The patch has been
|
|
[pushed](https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=57e5e645010430b3d73f8c6a757d09f48dc8f8d5),
|
|
and is also present at the latest version of Fedora GDB for Rawhide.
|
|
It wasn't possible to write a self-contained testcase for this
|
|
problem, so I had to resort to using an external tool (`eu-unstrip`)
|
|
in order to guarantee that the build-id note is correctly present in
|
|
the corefile. But that's a small detail, of course.
|
|
|
|
Anyway, I hope this was an interesting (albeit large) read!
|