blog/content/posts/gdb-and-systemtap-probes-part-3.md

---
date: 2012-11-02T00:00:00-05:00
title: "GDB and SystemTap Probes -- part 3"
tags: [en_us, english, gdb, systemtap, howto, fedora-planet, free-software]
---

Hi everybody :-).

I finally got some time to finish this series of posts, and I hope you
like the overall result. For those of you who are reading this blog
for the first time, you can access the first
post
[here]({filename}/2012-03-29-gdb-and-systemtap-probes-part-1.md), and
the
second
[here]({filename}/2012-10-27-gdb-and-systemtap-probes-part-2.md).

My goal with this third post is to talk a little bit about how you can
use the `SDT` probes with `tracepoints` inside `GDB`. Maybe this
particular feature will not be so helpful to you, but I recommend
reading the post either way. I will also give a brief explanation about
how the `SDT` probes are laid out inside the binary. So, let's start!

Complementary information
-------------------------

In my last post, I forgot to mention that the `SDT` probe support
present on older versions of Fedora `GDB` is not exactly as the way I
described here. This is because Fedora `GDB` adopted this feature much
earlier than upstream `GDB` itself, so while this has a great positive
aspect in terms of how the distro's philosophy works (i.e., Fedora
contains leading-edge features, so if you want to know how to FLOSS
community will be in a few months, use it!), it also has the downside of
delivering older/different versions of features in older Fedoras. But of
course, this `SDT` feature will be fully available on Fedora 18, to be
announced soon.

My suggestion is that if you use a not-so-recent Fedora (like Fedora 16,
15, etc), please upgrade it to the last version, or compile your own
version of `GDB` yourself (it's not that hard, I will make a post about
it in the next days/weeks!).

With that said, let's move on to our main topic here.

SDT Probes and Tracepoint
-------------------------

Before anything else, let me explain what a `tracepoint` is. Think of it
as *a breakpoint which doesn't stop the program's execution
when it hits*. In fact, it's a bit more than that: you can define
**actions** associated with a `tracepoint`, and those actions will be
performed when the `tracepoint` is hit. Neat, huh? :-)

There is a nice description of what a `tracepoint` in the [GDB
documentation](http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html#Tracepoints),
I recommend you give it a reading to understand the concept.

Ok, so now we have to learn how to put `tracepoints` in our code, and
how to define actions for them. But before that, let's remember our
example program:

```c
#include <sys/sdt.h>

int
main (int argc, char *argv[])
{
	int a = 10;

    STAP_PROBE1 (test_program, my_probe, a);

 	return 0;
}
```

Very simple, isn't it? Ok, to the `tracepoints` now, my friends.

Using `tracepoints` inside `GDB`
--------------------------------

In order to properly use `tracepoints` inside `GDB`, you will need to
use `gdbserver`, a tiny version of `GDB` suitable for debugging programs
remotely, over the net or serial line. In short, this is because `GDB`
cannot put tracepoints on a program running directly under it, so we
have to run it inside `gdbserver` and then connect `GDB` to it.

### Running our program inside `gdbserver`

In our case, we will just start `gdbserver` in our machine, order it to
listen to some high port, and connect to it through `localhost`, so
there will be no need to have access to another computer or device.

First of all, make sure you have `gdbserver` installed. If you use
Fedora, the package name you will have to install is `gdb-gdbserver`. If
you have it installed, you can do:

```console
$ gdbserver :3001 ./test_program
Process ./test_program created; pid = 17793
Listening on port 3001
```

The second argument passed to `gdbserver` instructs it to listen on the
port 3001 of your loopback interface, a.k.a. `localhost`.

You will notice that `gdbserver` will stay there indefinitely, waiting
for new connections to arrive. Don't worry, we will connect to it soon!

### Connecting an instance of `GDB` to `gdbserver`

Now, go to another terminal and start `GDB` with our program:

```console
$ gdb ./test_program
...
(gdb) target remote :3001
Remote debugging using :3001
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x0000003d60401530 in _start () from /lib64/ld-linux-x86-64.so.2
```

The command you have to use inside `GDB` is `target remote`. It takes as
an argument the host and the port to which you want to connect. In our
case, we just want it to connect to `localhost`, port 3001. If you saw
an output like the above, great, things are working for you (*don't pay
attention to the messages about
glibc debug information*). If you didn't see it, please check to see if
you're connecting to the right port, and if no other service is using
it.

Ok, so now it is time to start our *trace experiment*!

### Creating the `tracepoints`

**Every command should be issued on GDB, not on gdbserver!**

In your `GDB` prompt, put a `tracepoint` in the probe named `my_probe`:

```console
(gdb) trace -probe-stap my_probe
Tracepoint 1 at 0x4005a9
```

As you can see, the `trace` command takes exactly the same arguments as
the `break` command. Thus, you need to use the `-probe-stap` modified in
order to instruct `GDB` to put the `tracepoint` in the probe.

And now, let's define the **actions** associated with this `tracepoint`.
To do that, we use the `actions` command, which is an interactive
command inside `GDB`. It takes some specific keywords, and if you want
to learn more about it, please take a look at [this
link](http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoint-Actions.html#Tracepoint-Actions).
For this example, we will use only the `collect` keyword, which tells
`GDB` to... hm... collect something :-). In our case, it will collect
the probe's first argument, or `$_probe_arg0`, as you may remember.

```console
(gdb) actions 
Enter actions for tracepoint 1, one per line.
End with a line saying just "end".
>collect $_probe_arg0
>end
(gdb)
```

Simple as that. Finally, we have to define a `breakpoint` in the last
instruction of our program, because it is necessary to keep it running
on `gdbserver` in order to examine the `tracepoints` later. If we didn't
put this `breakpoint`, our program would finish and `gdbserver` would
not be able to provide information about what happened with our
`tracepoints`. In our case, we will simply put a `breakpoint` on line
10, i.e., on the `return 0;`:

### Running the trace experiment

Ok, time to run our trace experiment. First, we must issue a `tstart` to
tell `GDB` to start monitoring the `tracepoints`. And then, we can
continue our program normally.

```console
(gdb) tstart 
(gdb) continue
Continuing.

Breakpoint 1, main (argc=1, argv=0x7fffffffde88) at /tmp/test_program.c:10
10        return 0;
(gdb) tstop
(gdb)
```

Remember, `GDB` is **not** going to stop your program, because
`tracepoints` are designed to not interfere with the execution of it.
Also notice that we have also stopped the trace experiment after the
`breakpoint` hit, by using the `tstop` command.

Now, we will be able to examine what the `tracepoint` has collected.
First, we will the `tfind` command to make sure the `tracepoint` has
hit, and then we can inspect what we ordered it to collect:

```console
(gdb) tfind start
Found trace frame 0, tracepoint 1
8         STAP_PROBE1 (test_program, my_probe, a);
(gdb) p $_probe_arg0
$1 = 10
```

And it works! Notice that we are printing the probe argument using the
same notation as with `breakpoints`, even though we are not exactly
executing the `STAP_PROBE1` instruction. What does it mean? Well, with
the `tfind start` command we tell `GDB` to actually use the trace frame
collected during the program's execution, which, in this case, is the
probe argument. If you know `GDB`, think of it as if we were using the
`frame` command to jump back to a specific frame, where we would have
access to its state.

This is a very simple example of how to use the `SDT` probe support in
`GDB` with `tracepoints`. There is much more you can do, but I hope I
could explain the basics so that you can start playing with this
feature.

How the `SDT` probe is laid out in the binary
---------------------------------------------

You might be interested in learning how the probes are created inside
the binary. Other than reading the source code of
`/usr/include/sys/sdt.h`, which is the heart of the whole feature, I
also recommend [this
page](http://sourceware.org/systemtap/wiki/UserSpaceProbeImplementation),
which explains in detail what's going on under the hood. I also
recommend that you study a little about how the ELF format works,
specifically about notes in the ELF file.

Conclusion
----------

After this series of blog posts, I expect that you will now be able to
use the not-so-new feature of `SDT` probe support on `GDB`. Of course,
if you find some bug while using this, please feel free to report it
using [our bugzilla](http://sourceware.org/bugzilla/). And if you have
some question, use the comment system below and I will answer ASAP :-).

See ya, and thanks for reading!