238 lines
9 KiB
Markdown
238 lines
9 KiB
Markdown
---
|
|
date: 2012-11-02T00:00:00-05:00
|
|
title: "GDB and SystemTap Probes -- part 3"
|
|
tags: [en_us, gdb, systemtap, howto, fedora-planet, free-software]
|
|
---
|
|
|
|
Hi everybody :-).
|
|
|
|
I finally got some time to finish this series of posts, and I hope you
|
|
like the overall result. For those of you who are reading this blog
|
|
for the first time, you can access the first
|
|
post
|
|
[here]({filename}/2012-03-29-gdb-and-systemtap-probes-part-1.md), and
|
|
the
|
|
second
|
|
[here]({filename}/2012-10-27-gdb-and-systemtap-probes-part-2.md).
|
|
|
|
My goal with this third post is to talk a little bit about how you can
|
|
use the `SDT` probes with `tracepoints` inside `GDB`. Maybe this
|
|
particular feature will not be so helpful to you, but I recommend
|
|
reading the post either way. I will also give a brief explanation about
|
|
how the `SDT` probes are laid out inside the binary. So, let's start!
|
|
|
|
Complementary information
|
|
-------------------------
|
|
|
|
In my last post, I forgot to mention that the `SDT` probe support
|
|
present on older versions of Fedora `GDB` is not exactly as the way I
|
|
described here. This is because Fedora `GDB` adopted this feature much
|
|
earlier than upstream `GDB` itself, so while this has a great positive
|
|
aspect in terms of how the distro's philosophy works (i.e., Fedora
|
|
contains leading-edge features, so if you want to know how to FLOSS
|
|
community will be in a few months, use it!), it also has the downside of
|
|
delivering older/different versions of features in older Fedoras. But of
|
|
course, this `SDT` feature will be fully available on Fedora 18, to be
|
|
announced soon.
|
|
|
|
My suggestion is that if you use a not-so-recent Fedora (like Fedora 16,
|
|
15, etc), please upgrade it to the last version, or compile your own
|
|
version of `GDB` yourself (it's not that hard, I will make a post about
|
|
it in the next days/weeks!).
|
|
|
|
With that said, let's move on to our main topic here.
|
|
|
|
SDT Probes and Tracepoint
|
|
-------------------------
|
|
|
|
Before anything else, let me explain what a `tracepoint` is. Think of it
|
|
as *a breakpoint which doesn't stop the program's execution
|
|
when it hits*. In fact, it's a bit more than that: you can define
|
|
**actions** associated with a `tracepoint`, and those actions will be
|
|
performed when the `tracepoint` is hit. Neat, huh? :-)
|
|
|
|
There is a nice description of what a `tracepoint` in the [GDB
|
|
documentation](http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html#Tracepoints),
|
|
I recommend you give it a reading to understand the concept.
|
|
|
|
Ok, so now we have to learn how to put `tracepoints` in our code, and
|
|
how to define actions for them. But before that, let's remember our
|
|
example program:
|
|
|
|
#!c
|
|
#include <sys/sdt.h>
|
|
|
|
int
|
|
main (int argc, char *argv[])
|
|
{
|
|
int a = 10;
|
|
|
|
STAP_PROBE1 (test_program, my_probe, a);
|
|
|
|
return 0;
|
|
}
|
|
|
|
|
|
Very simple, isn't it? Ok, to the `tracepoints` now, my friends.
|
|
|
|
Using `tracepoints` inside `GDB`
|
|
--------------------------------
|
|
|
|
In order to properly use `tracepoints` inside `GDB`, you will need to
|
|
use `gdbserver`, a tiny version of `GDB` suitable for debugging programs
|
|
remotely, over the net or serial line. In short, this is because `GDB`
|
|
cannot put tracepoints on a program running directly under it, so we
|
|
have to run it inside `gdbserver` and then connect `GDB` to it.
|
|
|
|
### Running our program inside `gdbserver`
|
|
|
|
In our case, we will just start `gdbserver` in our machine, order it to
|
|
listen to some high port, and connect to it through `localhost`, so
|
|
there will be no need to have access to another computer or device.
|
|
|
|
First of all, make sure you have `gdbserver` installed. If you use
|
|
Fedora, the package name you will have to install is `gdb-gdbserver`. If
|
|
you have it installed, you can do:
|
|
|
|
#!bash
|
|
$ gdbserver :3001 ./test_program
|
|
Process ./test_program created; pid = 17793
|
|
Listening on port 3001
|
|
|
|
The second argument passed to `gdbserver` instructs it to listen on the
|
|
port 3001 of your loopback interface, a.k.a. `localhost`.
|
|
|
|
You will notice that `gdbserver` will stay there indefinitely, waiting
|
|
for new connections to arrive. Don't worry, we will connect to it soon!
|
|
|
|
### Connecting an instance of `GDB` to `gdbserver`
|
|
|
|
Now, go to another terminal and start `GDB` with our program:
|
|
|
|
#!bash
|
|
$ gdb ./test_program
|
|
...
|
|
(gdb) target remote :3001
|
|
Remote debugging using :3001
|
|
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
|
|
Loaded symbols for /lib64/ld-linux-x86-64.so.2
|
|
0x0000003d60401530 in _start () from /lib64/ld-linux-x86-64.so.2
|
|
|
|
The command you have to use inside `GDB` is `target remote`. It takes as
|
|
an argument the host and the port to which you want to connect. In our
|
|
case, we just want it to connect to `localhost`, port 3001. If you saw
|
|
an output like the above, great, things are working for you (*don't pay
|
|
attention to the messages about
|
|
glibc debug information*). If you didn't see it, please check to see if
|
|
you're connecting to the right port, and if no other service is using
|
|
it.
|
|
|
|
Ok, so now it is time to start our *trace experiment*!
|
|
|
|
### Creating the `tracepoints`
|
|
|
|
**Every command should be issued on GDB, not on gdbserver!**
|
|
|
|
In your `GDB` prompt, put a `tracepoint` in the probe named `my_probe`:
|
|
|
|
#!bash
|
|
(gdb) trace -probe-stap my_probe
|
|
Tracepoint 1 at 0x4005a9
|
|
|
|
As you can see, the `trace` command takes exactly the same arguments as
|
|
the `break` command. Thus, you need to use the `-probe-stap` modified in
|
|
order to instruct `GDB` to put the `tracepoint` in the probe.
|
|
|
|
And now, let's define the **actions** associated with this `tracepoint`.
|
|
To do that, we use the `actions` command, which is an interactive
|
|
command inside `GDB`. It takes some specific keywords, and if you want
|
|
to learn more about it, please take a look at [this
|
|
link](http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoint-Actions.html#Tracepoint-Actions).
|
|
For this example, we will use only the `collect` keyword, which tells
|
|
`GDB` to... hm... collect something :-). In our case, it will collect
|
|
the probe's first argument, or `$_probe_arg0`, as you may remember.
|
|
|
|
#!bash
|
|
(gdb) actions
|
|
Enter actions for tracepoint 1, one per line.
|
|
End with a line saying just "end".
|
|
>collect $_probe_arg0
|
|
>end
|
|
(gdb)
|
|
|
|
Simple as that. Finally, we have to define a `breakpoint` in the last
|
|
instruction of our program, because it is necessary to keep it running
|
|
on `gdbserver` in order to examine the `tracepoints` later. If we didn't
|
|
put this `breakpoint`, our program would finish and `gdbserver` would
|
|
not be able to provide information about what happened with our
|
|
`tracepoints`. In our case, we will simply put a `breakpoint` on line
|
|
10, i.e., on the `return 0;`:
|
|
|
|
### Running the trace experiment
|
|
|
|
Ok, time to run our trace experiment. First, we must issue a `tstart` to
|
|
tell `GDB` to start monitoring the `tracepoints`. And then, we can
|
|
continue our program normally.
|
|
|
|
#!bash
|
|
(gdb) tstart
|
|
(gdb) continue
|
|
Continuing.
|
|
|
|
Breakpoint 1, main (argc=1, argv=0x7fffffffde88) at /tmp/test_program.c:10
|
|
10 return 0;
|
|
(gdb) tstop
|
|
(gdb)
|
|
|
|
Remember, `GDB` is **not** going to stop your program, because
|
|
`tracepoints` are designed to not interfere with the execution of it.
|
|
Also notice that we have also stopped the trace experiment after the
|
|
`breakpoint` hit, by using the `tstop` command.
|
|
|
|
Now, we will be able to examine what the `tracepoint` has collected.
|
|
First, we will the `tfind` command to make sure the `tracepoint` has
|
|
hit, and then we can inspect what we ordered it to collect:
|
|
|
|
#!bash
|
|
(gdb) tfind start
|
|
Found trace frame 0, tracepoint 1
|
|
8 STAP_PROBE1 (test_program, my_probe, a);
|
|
(gdb) p $_probe_arg0
|
|
$1 = 10
|
|
|
|
And it works! Notice that we are printing the probe argument using the
|
|
same notation as with `breakpoints`, even though we are not exactly
|
|
executing the `STAP_PROBE1` instruction. What does it mean? Well, with
|
|
the `tfind start` command we tell `GDB` to actually use the trace frame
|
|
collected during the program's execution, which, in this case, is the
|
|
probe argument. If you know `GDB`, think of it as if we were using the
|
|
`frame` command to jump back to a specific frame, where we would have
|
|
access to its state.
|
|
|
|
This is a very simple example of how to use the `SDT` probe support in
|
|
`GDB` with `tracepoints`. There is much more you can do, but I hope I
|
|
could explain the basics so that you can start playing with this
|
|
feature.
|
|
|
|
How the `SDT` probe is laid out in the binary
|
|
---------------------------------------------
|
|
|
|
You might be interested in learning how the probes are created inside
|
|
the binary. Other than reading the source code of
|
|
`/usr/include/sys/sdt.h`, which is the heart of the whole feature, I
|
|
also recommend [this
|
|
page](http://sourceware.org/systemtap/wiki/UserSpaceProbeImplementation),
|
|
which explains in detail what's going on under the hood. I also
|
|
recommend that you study a little about how the ELF format works,
|
|
specifically about notes in the ELF file.
|
|
|
|
Conclusion
|
|
----------
|
|
|
|
After this series of blog posts, I expect that you will now be able to
|
|
use the not-so-new feature of `SDT` probe support on `GDB`. Of course,
|
|
if you find some bug while using this, please feel free to report it
|
|
using [our bugzilla](http://sourceware.org/bugzilla/). And if you have
|
|
some question, use the comment system below and I will answer ASAP :-).
|
|
|
|
See ya, and thanks for reading!
|