--- date: 2012-11-02T00:00:00-05:00 title: "GDB and SystemTap Probes -- part 3" tags: [en_us, english, gdb, systemtap, howto, fedora-planet, free-software] --- Hi everybody :-). I finally got some time to finish this series of posts, and I hope you like the overall result. For those of you who are reading this blog for the first time, you can access the first post [here]({filename}/2012-03-29-gdb-and-systemtap-probes-part-1.md), and the second [here]({filename}/2012-10-27-gdb-and-systemtap-probes-part-2.md). My goal with this third post is to talk a little bit about how you can use the `SDT` probes with `tracepoints` inside `GDB`. Maybe this particular feature will not be so helpful to you, but I recommend reading the post either way. I will also give a brief explanation about how the `SDT` probes are laid out inside the binary. So, let's start! Complementary information ------------------------- In my last post, I forgot to mention that the `SDT` probe support present on older versions of Fedora `GDB` is not exactly as the way I described here. This is because Fedora `GDB` adopted this feature much earlier than upstream `GDB` itself, so while this has a great positive aspect in terms of how the distro's philosophy works (i.e., Fedora contains leading-edge features, so if you want to know how to FLOSS community will be in a few months, use it!), it also has the downside of delivering older/different versions of features in older Fedoras. But of course, this `SDT` feature will be fully available on Fedora 18, to be announced soon. My suggestion is that if you use a not-so-recent Fedora (like Fedora 16, 15, etc), please upgrade it to the last version, or compile your own version of `GDB` yourself (it's not that hard, I will make a post about it in the next days/weeks!). With that said, let's move on to our main topic here. SDT Probes and Tracepoint ------------------------- Before anything else, let me explain what a `tracepoint` is. Think of it as *a breakpoint which doesn't stop the program's execution when it hits*. In fact, it's a bit more than that: you can define **actions** associated with a `tracepoint`, and those actions will be performed when the `tracepoint` is hit. Neat, huh? :-) There is a nice description of what a `tracepoint` in the [GDB documentation](http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoints.html#Tracepoints), I recommend you give it a reading to understand the concept. Ok, so now we have to learn how to put `tracepoints` in our code, and how to define actions for them. But before that, let's remember our example program: ```c #include int main (int argc, char *argv[]) { int a = 10; STAP_PROBE1 (test_program, my_probe, a); return 0; } ``` Very simple, isn't it? Ok, to the `tracepoints` now, my friends. Using `tracepoints` inside `GDB` -------------------------------- In order to properly use `tracepoints` inside `GDB`, you will need to use `gdbserver`, a tiny version of `GDB` suitable for debugging programs remotely, over the net or serial line. In short, this is because `GDB` cannot put tracepoints on a program running directly under it, so we have to run it inside `gdbserver` and then connect `GDB` to it. ### Running our program inside `gdbserver` In our case, we will just start `gdbserver` in our machine, order it to listen to some high port, and connect to it through `localhost`, so there will be no need to have access to another computer or device. First of all, make sure you have `gdbserver` installed. If you use Fedora, the package name you will have to install is `gdb-gdbserver`. If you have it installed, you can do: ```console $ gdbserver :3001 ./test_program Process ./test_program created; pid = 17793 Listening on port 3001 ``` The second argument passed to `gdbserver` instructs it to listen on the port 3001 of your loopback interface, a.k.a. `localhost`. You will notice that `gdbserver` will stay there indefinitely, waiting for new connections to arrive. Don't worry, we will connect to it soon! ### Connecting an instance of `GDB` to `gdbserver` Now, go to another terminal and start `GDB` with our program: ```console $ gdb ./test_program ... (gdb) target remote :3001 Remote debugging using :3001 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 0x0000003d60401530 in _start () from /lib64/ld-linux-x86-64.so.2 ``` The command you have to use inside `GDB` is `target remote`. It takes as an argument the host and the port to which you want to connect. In our case, we just want it to connect to `localhost`, port 3001. If you saw an output like the above, great, things are working for you (*don't pay attention to the messages about glibc debug information*). If you didn't see it, please check to see if you're connecting to the right port, and if no other service is using it. Ok, so now it is time to start our *trace experiment*! ### Creating the `tracepoints` **Every command should be issued on GDB, not on gdbserver!** In your `GDB` prompt, put a `tracepoint` in the probe named `my_probe`: ```console (gdb) trace -probe-stap my_probe Tracepoint 1 at 0x4005a9 ``` As you can see, the `trace` command takes exactly the same arguments as the `break` command. Thus, you need to use the `-probe-stap` modified in order to instruct `GDB` to put the `tracepoint` in the probe. And now, let's define the **actions** associated with this `tracepoint`. To do that, we use the `actions` command, which is an interactive command inside `GDB`. It takes some specific keywords, and if you want to learn more about it, please take a look at [this link](http://sourceware.org/gdb/current/onlinedocs/gdb/Tracepoint-Actions.html#Tracepoint-Actions). For this example, we will use only the `collect` keyword, which tells `GDB` to... hm... collect something :-). In our case, it will collect the probe's first argument, or `$_probe_arg0`, as you may remember. ```console (gdb) actions Enter actions for tracepoint 1, one per line. End with a line saying just "end". >collect $_probe_arg0 >end (gdb) ``` Simple as that. Finally, we have to define a `breakpoint` in the last instruction of our program, because it is necessary to keep it running on `gdbserver` in order to examine the `tracepoints` later. If we didn't put this `breakpoint`, our program would finish and `gdbserver` would not be able to provide information about what happened with our `tracepoints`. In our case, we will simply put a `breakpoint` on line 10, i.e., on the `return 0;`: ### Running the trace experiment Ok, time to run our trace experiment. First, we must issue a `tstart` to tell `GDB` to start monitoring the `tracepoints`. And then, we can continue our program normally. ```console (gdb) tstart (gdb) continue Continuing. Breakpoint 1, main (argc=1, argv=0x7fffffffde88) at /tmp/test_program.c:10 10 return 0; (gdb) tstop (gdb) ``` Remember, `GDB` is **not** going to stop your program, because `tracepoints` are designed to not interfere with the execution of it. Also notice that we have also stopped the trace experiment after the `breakpoint` hit, by using the `tstop` command. Now, we will be able to examine what the `tracepoint` has collected. First, we will the `tfind` command to make sure the `tracepoint` has hit, and then we can inspect what we ordered it to collect: ```console (gdb) tfind start Found trace frame 0, tracepoint 1 8 STAP_PROBE1 (test_program, my_probe, a); (gdb) p $_probe_arg0 $1 = 10 ``` And it works! Notice that we are printing the probe argument using the same notation as with `breakpoints`, even though we are not exactly executing the `STAP_PROBE1` instruction. What does it mean? Well, with the `tfind start` command we tell `GDB` to actually use the trace frame collected during the program's execution, which, in this case, is the probe argument. If you know `GDB`, think of it as if we were using the `frame` command to jump back to a specific frame, where we would have access to its state. This is a very simple example of how to use the `SDT` probe support in `GDB` with `tracepoints`. There is much more you can do, but I hope I could explain the basics so that you can start playing with this feature. How the `SDT` probe is laid out in the binary --------------------------------------------- You might be interested in learning how the probes are created inside the binary. Other than reading the source code of `/usr/include/sys/sdt.h`, which is the heart of the whole feature, I also recommend [this page](http://sourceware.org/systemtap/wiki/UserSpaceProbeImplementation), which explains in detail what's going on under the hood. I also recommend that you study a little about how the ELF format works, specifically about notes in the ELF file. Conclusion ---------- After this series of blog posts, I expect that you will now be able to use the not-so-new feature of `SDT` probe support on `GDB`. Of course, if you find some bug while using this, please feel free to report it using [our bugzilla](http://sourceware.org/bugzilla/). And if you have some question, use the comment system below and I will answer ASAP :-). See ya, and thanks for reading!