Tags
C++, corefile, Debugging, ELF, Google Core Dumper, memory map, pmap, readelf
I am quite proud to say that, I am a Debug Enthusiast . People like me believe in the below quote…
Debugging is an art…
Debugging can sometimes be hell. Like roaming around a maze in final challenge of Triwizard tournament or like searching for needle of unknown size and shape in pacific ocean.
Being an debug enthusiast, I spend my time exploring ways to better my debugging skills and also to reach out to vent my learning to folks like you [who stop by @ my blog… 😉 ].
One such new learning, which I had, recent days is GoogleCoreDumper.
It is a wonderful tool, which aids you in creating a GDB readable ELF corefile. Most important aspect is, Coredump can be dumped, without stopping your application.
This is normal to me… as I am used to gcore [ however googCoreDumper might be helpful in systems where ulimit setting for writing a core is an issue. For more details.. look here]
So.. what’s special about googCore according to me….????? 😉 😛
Writing customized notes in the corefile…
Before understanding customized notes, let us first understand what are notes in corefiles…
ELF notes are special information, which will get stored in the ELF header. ELF as the name suggests, is a popular file format, opted by most OS vendors to convey the information in uniform manner.
An Executable file [exe ], coredump etc are ELF files. ELF note appear in ELF header [metadata of the ELF file ] which conveys vendor & platform specific information like..
- OS platform
- Signal name which caused the core
- Executable name for which the core belongs to
- Endianess hardware etc…
I am not going to bore you with the theory anymore. This link has got much info, incase if you are interested to learn more about ELF.
Predominantly these notes are vendor specific and cannot be customized unless you keep your hands in kernel and tweak something. Though there were some ELF writers like elfIO, elfutils etc.. the simplicity and usability doesn’t match with GoogleCoreDumper (according to me 😛 🙂 ).
In this post, let me take you through, how a typical debugging problem could be solved by using GoogleCoreDumper esp on the custom notes…
Problem : How to get pmap [process memory map] from a linux coredump , where kernel < 3.6
I work in a server product, which is a c++ app and crashdumps from deployment site are the primary source of information for me to debug. One of the pains in debugging is to rule-out theories based on some proof of concept. This product employ both proprietary plugins and 3rd party / opensource plugins, all hosted in a same process space.
If a crash / core occurs in deployment site with the release mode binaries , all I have is just the function addresses in hex format. As I had explained earlier, my first job is to understand whether the crashed instruction is part of proprietary plugin [ ha… I know where to look @ 😉 ] or opensource [ well… some what lucky… :)] or 3rd party [phew!!! this is not in my court…. 🙂 ].
So how do I get to know.. what region of memory space is occupied by every shared library in the process space… the answer is pmap.
However pmap utility will work only with the running process and not with core.
Well.. wouldn’t the crash capture the memory map as part of core ???
I can read your mind…. 😉
The answer is, until this check-in the memory map of the running process, was not captured in crashdump. In new kernel this has been resolved, but what about old kernel… [like my situation… 😦 ].
Now read the blog title… that’s it.. you have the solution…
Implementation details
GoogleCoreDumper can be used to both read & write coredumps. It also provides facility to add custom notes in the corefile through
SetCoreDumpNotes()
api . Surprsingly, these functions aren’t explained in the project wiki page. Nevertheless the man pages as part of package was still useful.
SetCoreDumpNotes() api function sets the attributes in params to add additional notes to core dumps when called with GetCoreDumpWith() or WriteCoreDumpWith()
The notes are specified as an array of the Core‐dumperNote structure:
struct CoredumperNote { const char *name; // The vendor name unsigned int type; // A vendor specific type unsigned int description_size; // The size of the description field const void *description; // The note data };
The name, including a terminating null character, will be 4 byte
aligned. The type is a user chosen value. The description_size speci‐
fies the amount of bytes to write from the description pointer in mem‐
ory to the core dump note. The description will be padded to be 4 byte aligned.
You can dump as many information you want, provided the total corefile size doesn’t exceed the ulimit setting of the system.
Code
The intent of this exercise is to capture the memory map of the process while writing the coredump file and to extract the same from the resultant core file.
Part 1 – Saving information in the corefile.
First a signal handler is registered to capture the signal and being the core writing algorithm
int main() { signal(SIGSEGV, signal_handler); cout << "Starting the program with pid" << getpid() << endl; int c; cout << "OK am now waiting for the signal" << endl; while(1); return 0; }
and now we will see, what the signal_handler is actually doing..
/* * googCore.cpp * * Author: viswa */ #include <stdio.h> #include <stdlib.h> #include <string.h> #include <google/coredumper.h> // Header for googCore API #include <signal.h> // for handling signal #include <iostream> #include <fstream> #include <sys/types.h> #include <unistd.h> #include <sys/stat.h> #include <fcntl.h> #include <sys/mman.h> #include <sstream> #include <limits> #include <errno.h> using namespace std; #define NT_VISWA 0x46494c46 // NT_FILE + 1 void signal_handler(int SIGNO) { : // Jumping to relevant details... : // now lets proceed to dump the core struct CoreDumpParameters theParam; struct CoredumperNote theNote; theNote.name = "kspviswa-special-notes-mmap"; theNote.type = NT_VISWA; theNote.description_size = fsize; theNote.description = pInit; // pInit is the ptr to pmap of the running process // Clear theParam so that default values get initialized ClearCoreDumpParameters(&theParam); // Save theParam with the custom notes SetCoreDumpNotes(&theParam, &theNote, 1); // Write the notes to core status = WriteCoreDumpWith(&theParam, "viswa-core");
The NT_VISWA is a special flag which helps to uniquely identify the special notes in the elfReader.
Part 2 – Reading special notes from the corefile.
In this exercise, I had used readelf to customize / tweak the reader so that special notes can be read [ I love opensource extremely for these customization 😉 ] .
static const char * get_note_type (unsigned e_type) { : /** * ########## VISWA ############# */ case NT_VISWA: return _("NT_VISWA (kspviswa-special-notes-mmap)"); } /** * ########## VISWA ############# */ static int print_core_note_special(Elf_Internal_Note *pnote) { printf("\n Printing the special notes for kspviswa... :-)\n"); printf("%s",(char*) pnote->descdata); return 1; } static int process_note (Elf_Internal_Note * pnote) { : /** * ########## VISWA ############# */ else if (const_strneq (pnote->namedata, "kspviswa-special-notes-mmap")) return print_core_note_special (pnote); else return 1; }
Results
As you can see, the pmap output taken just before the core and the pmap saved as notes in the core, do matches. That’s it… Now the readelfviswa is able to read the pmap from core. Problem solved 🙂
Sourcecode Download
I had opened a github account recently and googCore is my first contribution of source code to opensource via github. You can visit the source here.
Just before waving you goodbye….
GoogleCoreDumper was extremely cool and fun to work with. Solving a realworld problem with my hobby was lot more fantastic. Not only just pmap, any interesting information like memory profiling, number of active connections to a shared resource, total count of shared libraries, state of global caches / buffers, total number of heap space contributed by a particular module etc can be logged as notes to the corefile and all you need to do is to tweak readelf to read those for you… I enjoy debugging 🙂 Hope you enjoyed my blog too…
Once again, thanks for stopping @ my blog and took time to go through this post. Feel free to leave down your comments / feedbacks and I will be more than willing to respond at the earliest.
Happy Coding [& Debugging 🙂 ] !!!!!