Memory allocation debugging with glibc – tracing

Although there are many great tools to debug memory allocation errors, like Valgrind or Electric Fence, sometimes these tools aren’t available or it’s not feasible to use them. In this post I want to show you a  debugging technique that doesn’t require any other software beside GNU C library. All of the examples were created and ran on a standard x86-64 machine.

GNU C library provides malloc(), free() and other related routines for dynamic memory allocation. Alongside malloc() and friends, glibc features two very interesting mechanisms to help finding common dynamic memory allocation errors. The one I want to present to you is the memory allocation tracing.

Tracing memory allocation calls

Glibc features a tracing functionality for memory allocation debugging. It is enabled by setting a special environment variable and using mtrace() function.

The mtrace() function modifies the behavior of malloc functions family by installing hooks for malloc(), realloc() and free() functions, making all the calls to these routines traced and logged. MALLOC_TRACE environment variable should contain a valid file name to which the tracing data is written. If the file already exists it is overwritten (truncated). If the variable is not defined or does not contain a valid file path (i.e. file couldn’t be opened for writing) tracing functionality is disabled and no hooks are installed. The muntrace() function disables malloc tracing altogether. For security reasons, tracing is automatically disabled by the dynamic runtime linker for set-user-ID and set-group-ID programs.

Here’s an example program:

As you can see, this code allocates some memory and doesn’t free it. Of course, this is easy to spot in a straightforward example  like this. Moreover, operating system will free all process’s resources on termination, so it’s no big deal after all. However, real life cases are almost never so obvious ;). What if this code is a part of a daemon that is supposed to run non-stop for months or years? So, let’s see how to use malloc tracing on this:

Here, mtrace() and muntrace() will only be compiled when DEBUG_ON symbol is defined. It is a good practice because tracing will slow down malloc() functions a bit. Now, lets compile the file…

…and set the MALLOC_TRACE environment variable with a file name

After executing this program there should be a file named “tracefile” present in the current directory. Although it’s a plain text file, it’s really not too readable. However, the mtrace tool script is intended to parse it and make it human-readable. Here’s how it looks on my machine:

We have a (potential) memory leak! However, we only have an address in the Caller collumn – not really a convenient way to track the leak, is it? The mtrace script can provide more meaningful information. Executing mtrace script with the binary name and a trace file name will result in this:

There we go! We have pinpointed an potentially non-freed memory allocation. Memory allocation tracing can also be switched on/off on demand. If the MALLOC_TRACE is set properly, you can for example use signals to turn tracing on and off, like presented in the Glibc manual:

The Glibc manual suggests that calling muntrace() before the program termination is generally not a good idea. The libraries  can’t free the memory they use before the application returns from main(). Therefore, the best we can do is to call mtrace() as one of the very first calls in the application and never call muntrace().

Tracing without changing your application

There is a way to enable memory allocation tracing even without modyfing and recompiling the application. It will allow to trace all of the malloc calls for the application AND its libraries as well (including the calls in application and library constructors).Here’s how it goes:

First, we have to create a shared object – a library that is dynamically loaded when it’s needed. In this library we will use gcc attributes for marking two functions as the lib’s constructor and destructor, which are called upon loading and unloading the library.  This is the library code

Then, we compile it as a relocatable library…

… and make the “libtracelib.so” shared object out of it:

We have just created a libtracelib.so file – a dynamically loaded library. Then, we are going to use one of the dynamic linker environment variables to load this library before starting up our application – we use the code from the first listing as there are no mtrace and muntrace calls:

If the MALLOC_TRACE environment variable was set properly, we may now parse the created tracefile:

See? Quite simple and yet quite effective way to pinpoint potential memory leaks without modification to your application. Of course, it will work better if the application is compiled with debugging info, but, still, we have found a potential memory leak only by executing the app with a special preloaded library. Note that the SUID and SGID restrictions still apply in this case.

Please, feel free to comment the article (and the other ones as well).

Stay tuned for the next article – the heap consistency check with glibc.

Cheers! 😉
Cristos

2 thoughts on “Memory allocation debugging with glibc – tracing

  • 10th November 2014 at 12:01
    Permalink

    Finally some programming stuff! Waiting for next installments! 🙂

    Reply
    • 10th November 2014 at 13:11
      Permalink

      Thanks! Stay tuned, mate. More stuff is coming soon 😉

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload CAPTCHA.