Clocksource tsc unstable

The reason of the below message 2013-07-16T05:00:05.181538-04:00 xxxxxx kernel: Clocksource tsc unstable (delta = -95170507948 ns). Enable clocksource failover by adding clocksource_failover kernel parameter. clocksource structure /** * struct clocksource - hardware abstraction for a free running counter * Provides mostly state-free accessors to the underlying hardware. * This is the structure used for system… Continue reading Clocksource tsc unstable

pycrashext – A rich python extension

Based on Pykdump, I wrote a set of plugins named 'pycrashext' which is basically trying to help to reduce the troubleshooting time.  My favorite command in this set is 'edis' which can display source code in between disassembled lines. This requires an additional source code server with source codes, but once you have it, it… Continue reading pycrashext – A rich python extension

How to disassemble a module from a vmcore

There are times that you have to deal with a module which you don't have source code. Only thing we can do is disassemble it, but if you don't have actual module binary, this is also tough. Luckily, vmcore has all the code loaded into the memory. So, here's the steps to get disassembled code… Continue reading How to disassemble a module from a vmcore

How dump trace is generated in Linux kernel

Note for myself to remember how call trace is generated in Kernel /* * The architecture-independent dump_stack generator */ void dump_stack(void) { unsigned long stack; printk("Pid: %d, comm: %.20s %s %s %.*sn", current->pid, current->comm, print_tainted(), init_utsname()->release, (int)strcspn(init_utsname()->version, " "), init_utsname()->version); show_trace(NULL, NULL, &stack); } void show_trace(struct task_struct *task, struct pt_regs *regs, unsigned long *stack) {… Continue reading How dump trace is generated in Linux kernel

Jump into vmcore analysis – Step 8

There's a time you want to check the local variables or other entries in the stack. Below is an example that was crashed in 'kmem_freepages' and needed to check why it's crashed whiling freeing it. PID: 26 TASK: ffff81027f9197a0 CPU: 0 COMMAND: "events/0" #0 [ffff81027f92fa90] crash_kexec at ffffffff800aaa0c #1 [ffff81027f92fb50] __die at ffffffff8006520f #2 [ffff81027f92fb90]… Continue reading Jump into vmcore analysis – Step 8

Jump into vmcore analysis – Step 6

The real merit of the vmcore is that you can trace the code with the current value each variable holds. Here you can find one example that traces the filesystem which ended up with the corrupted data entry somehow. http://pagead2.googlesyndication.com/pagead/show_ads.js crash> bt PID: 6326 TASK: ffff810402165820 CPU: 1 COMMAND: "fuser" #0 [ffff8103b54efa80] crash_kexec at ffffffff800b099c… Continue reading Jump into vmcore analysis – Step 6