Sungju's Slow Life

As the kernel is running on top of all other services, it’s hard to debug it in a live system. You can use ‘gdb’ on a live system, but you only can check the current values of some exported symbols. You can’t use breakpoint on a running kernel. If you set a breakpoint, it’ll stop and you won’t be able to continue gdb as well.

So, here comes kprobes and jprobes. It doesn’t stop the kernel, but provide hooking mechanism to intercept when specific instruction is going to be called. Initially it was developed by IBM and named dprobes (Dynamic Probes), but now’s it’s named kprobes and included since 2.6.9 kernel (http://www.ibm.com/developerworks/library/l-kprobes.html). Red Hat website also has a good article regarding kprobes : http://www.redhat.com/magazine/005mar05/features/kprobes/. Actually, SystemTap uses kprobes internally. You might think why we need to learn how to use kprobes then. SystemTap provides more stable environment that doesn’t crash the system by not well implemented scripts and give you feeling it’s a script rather than something hard to understand kernel thing. The issue is there’s a time that you need to run SystemTap script in ‘guru’ mode as you need to access symbols that isn’t implemented in SystemTap yet. It causes of big complexity as it now has script and C code at the same page. In my opinion, in that case, pure kprobes module would be a better choice as it’s much cleaner.

You can check if kprobes is configured on your system by running the following. RHEL includes this feature by default.

$ grep "^CONFIG_KPROBES" /boot/config-2.6.32-358.6.1.el6.x86_64 
CONFIG_KPROBES=y

You can specify the handlers for the specific function to be called before, after, and for the exception. To register those handlers you can use the following kprobe structure which is located in include/linux/kprobes.h

struct kprobe {
        struct hlist_node hlist;
 
        /* list of kprobes for multi-handler support */
        struct list_head list;
 
        /* Indicates that the corresponding module has been ref counted */
        unsigned int mod_refcounted;
 
        /*count the number of times this probe was temporarily disarmed */
        unsigned long nmissed;
 
        /* location of the probe point */
        kprobe_opcode_t *addr;
 
        /* Allow user to indicate symbol name of the probe point */
        const char *symbol_name;
 
        /* Offset into the symbol */
        unsigned int offset;
 
        /* Called before addr is executed. */
        kprobe_pre_handler_t pre_handler;
 
        /* Called after addr is executed, unless... */
        kprobe_post_handler_t post_handler;
 
        /* ... called if executing addr causes a fault (eg. page fault).
         * Return 1 if it handled fault, otherwise kernel will see it. */
        kprobe_fault_handler_t fault_handler;
 
        /* ... called if breakpoint trap occurs in probe handler.
         * Return 1 if it handled break, otherwise kernel will see it. */
        kprobe_break_handler_t break_handler;
 
        /* Saved opcode (which has been replaced with breakpoint) */
        kprobe_opcode_t opcode;
 
        /* copy of the original instruction */
        struct arch_specific_insn ainsn;
};

Most important fields in the above structure are in below.

1. addr : Address of the instruction you want to monitor
2. symbol_name : You can use this symbol name instead of address if the symbol is exported
3. pre_handler : handler to be called before execute the instruction in addr/symbol_name
4. post_handler : handler to be called after executing the instruction in addr/symbol_name
5. fault_handler : handler to be called when there’s an fault while running addr/symbol_name. If this handler returns 1, the original fault handler won’t be called

To register and unregister kprobe handlers, you can use below functions.

int register_kprobe(struct kprobe *p);
void unregister_kprobe(struct kprobe *p);

Here’s an example that traces ‘copy_to_user()’ function call. Filename is kprobetest.c

#include 
#include 
#include 
#include 
#include 
 
static char *debug_func = "copy_to_user";
 
static struct kprobe my_kp;
 
static int handler_pre(struct kprobe *p, struct pt_regs *regs)
{
    printk("Before call : %sn", p->symbol_name);
    printk("at addr=0x%pn", p->addr);
    printk("   current process is %sn", current->comm);
    return 0;
}
 
static void handler_post(struct kprobe *p, struct pt_regs *regs,
             unsigned long flags)
{
    printk("After call : %sn", p->symbol_name);
    printk("at addr=0x%pn", p->addr);
    printk("   current process is %sn", current->comm);
}
 
static int handler_fault(struct kprobe *p, struct pt_regs *regs, 
             int trapnr)
{
    printk("Fault occured during call : %sn", p->symbol_name);
    printk("at addr=0x%pn", p->addr);
    printk("   current process is %sn", current->comm);
    return 0;
}
 
static int __init my_init(void)
{
    my_kp.pre_handler = handler_pre;
    my_kp.post_handler = handler_post;
    my_kp.fault_handler = handler_fault;
    my_kp.symbol_name = debug_func;
 
    if (register_kprobe(&my_kp))
        return -1;
 
    return 0;
}
 
static void __exit my_exit(void)
{
    unregister_kprobe(&my_kp);
}
 
module_init(my_init);
module_exit(my_exit);
 
MODULE_LICENSE("GPL");

Here’s the related Makefile

obj-m += kprobetest.o
 
export KROOT=/lib/modules/`uname -r`/build

allofit:   modules
 
modules:
        @$(MAKE) -C $(KROOT) M=$(PWD) modules
 
modules_install:
        @$(MAKE) -C $(KROOT) M=$(PWD) modules_install
 
clean:
        @rm -rf   *.o *.ko .*cmd *.mod.c .tmp_versions .*.d .*.tmp Module.symvers

To see how it works, you just need to load the module and check /var/log/messages.

$ make
make[1]: Entering directory `/usr/src/kernels/2.6.32-358.6.1.el6.x86_64'
  CC [M]  /root/Study/mysysrq.o
  CC [M]  /root/Study/kprobetest.o
  Building modules, stage 2.
  MODPOST 2 modules
  CC      /root/Study/kprobetest.mod.o
  LD [M]  /root/Study/kprobetest.ko.unsigned
  NO SIGN [M] /root/Study/kprobetest.ko
  CC      /root/Study/mysysrq.mod.o
  LD [M]  /root/Study/mysysrq.ko.unsigned
  NO SIGN [M] /root/Study/mysysrq.ko
make[1]: Leaving directory `/usr/src/kernels/2.6.32-358.6.1.el6.x86_64'
 
$ insmod kprobetest.ko
 
$ tail /var/log/messages 
May  5 19:19:32 localhost kernel:   current process is bash
May  5 19:19:32 localhost kernel: After call : copy_to_user
May  5 19:19:32 localhost kernel: at addr=0xffffffff81281ff0
May  5 19:19:32 localhost kernel:   current process is bash
May  5 19:19:32 localhost kernel: Before call : copy_to_user
May  5 19:19:32 localhost kernel: at addr=0xffffffff81281ff0
May  5 19:19:32 localhost kernel:   current process is bash
May  5 19:19:32 localhost kernel: After call : copy_to_user
May  5 19:19:32 localhost kernel: at addr=0xffffffff81281ff0
May  5 19:19:32 localhost kernel:   current process is bash
 
$ rmmod kprobetest