Friday, November 20, 2009

Profiling kernel code coverage



Measuring which lines of code get executed and how often can be a useful tool for debugging or testing. That capability has long been available for user space programs in the form of gcov. A recent patch seeks to allow kernel hackers access to the same tool.

There are three main components to making gcov work with the kernel: changing the build to add the -fprofile-arcs -ftest-coverage gcc flags, hooking up the gcc-generated code to record the coverage information, and providing a way for the kernel to output the data to user space. TheGCOV_PROFILE kconfig option governs whether to include gcov into the build, while GCOV_PROFILE_ALLactivates profiling for the entire kernel. If desired, individual directories and files can be selectively included or excluded from being instrumented.

The new kernel/gcov directory contains the necessary functions to support the gcc-generated profiling code. This includes handling statically linked kernel code as well as kernel modules that are loaded. Information gathered from code in modules can be either preserved or discarded when they are unloaded. This will allow analysis of the module unloading path that could be useful for detecting resource leaks or other problems in that process.

A user space program compiled for gcov will write a binary file to the filesystem for each source file that contains the data corresponding to the execution path through that file. The kernel needs to do that differently, so instead it writes to a file in debugfs. Each source file that is compiled for gcov will store its information in /sys/kernel/debug/gcov/path/file.gcda, where /sys/kernel/debug is the debugfs mount point and path is the path to the file in the kernel tree. The individual .gcda files can also be written to, which will result in setting the accumulated data for that source file back to zero.

Once the data has been gathered, gcov can be invoked to produce a file that annotates the source showing each line with the number of times it has been executed. LCOV is a graphical tool that can also be used to examine the coverage information. LCOV and the gcov kernel patches both come from the Linux Test Project which has an extensive kernel test suite and is using gcov to expand the coverage of their tests.

As part of the patch set, the seq_file interface has been extended to allow writing of arbitrary binary data to a virtual file. Currently, the seq_file interface is somewhat character oriented, so a function has been added to fs/seq_file.c to provide that ability:

    int seq_write(struct seq_file *seq, const void *data, size_t len) 
As the prototype implies, it writes len bytes from data to the seq_file seq.

Efforts to get gcov support into the kernel have been around since 2002, but the code was recently rewritten to be a better fit for recent kernels. In the patch, Peter Oberparleiter says "due to regular requests, I rewrote the gcov-kernel patch from scratch so that it would (hopefully) be fit for inclusion into the upstream kernel." One of the bigger changes is to move the user space interface for gcov from /procinto debugfs.

It seems that the technical issues have largely been addressed in the third version of the gcov patch. It can provide useful information, especially for increasing the reach of test coverage—something that can only help reduce kernel bugs—so it could make for a nice kernel addition. Whether it will be picked up into linux-next or -mm and pushed towards an eventual mainline merge remains to be seen.

Sunday, November 15, 2009

mutex

Mutexes may be recursively acquired, but they are intended to be held for a short period of time. Specifically, one may not sleep while holding a mutex. If you need to hold a lock across a sleep, use a lockmgr(9) lock

Monday, November 9, 2009

User Space vs Kernel Space

User Space vs Kernel Space

A kernel is all about access to resources, whether the resource in question happens to be a video card, a hard drive or even memory. Programs often compete for the same resource. As I just saved this document, updatedb started updating the locate database. My vim session and updatedb are both using the hard drive concurrently. The kernel needs to keep things orderly, and not give users access to resources whenever they feel like it. To this end, a CPU can run in different modes. Each mode gives a different level of freedom to do what you want on the system. The Intel 80386 architecture has 4 of these modes, which are called rings. Unix uses only two rings; the highest ring (ring 0, also known as `supervisor mode' where everything is allowed to happen) and the lowest ring, which is called `user mode'.

Recall the discussion about library functions vs system calls. Typically, you use a library function in user mode. The library function calls one or more system calls, and these system calls execute on the library function's behalf, but do so in supervisor mode since they are part of the kernel itself. Once the system call completes its task, it returns and execution gets transfered back to user mode.

The Linux Kernel Module Programming Guide

http://tldp.org/LDP/lkmpg/2.6/html/

Sunday, November 1, 2009

http://www.dumblittleman.com/2009/10/7-must-read-productivity-steps-to.html

Step #1: Write Down What You Want to Accomplish and Give it a Deadline
Step #2: Create a Simple Plan for Getting Things Done
Step #3: Schedule Critical Tasks Daily
Step #4: Eat that “Ugly Frog” First
Step #5: Focus on the task at Hand
Step #6: Just Say “No”
Step #7: Track and Report Your Progress