Clocksource tsc unstable

The reason of the below message 2013-07-16T05:00:05.181538-04:00 xxxxxx kernel: Clocksource tsc unstable (delta = -95170507948 ns). Enable clocksource failover by adding clocksource_failover kernel parameter. clocksource structure /** * struct clocksource – hardware abstraction for a free running counter * Provides mostly state-free accessors to the underlying hardware. * This is the structure used for systemContinue reading “Clocksource tsc unstable”

pycrashext – A rich python extension

Based on Pykdump, I wrote a set of plugins named ‘pycrashext’ which is basically trying to help to reduce the troubleshooting time.  My favorite command in this set is ‘edis’ which can display source code in between disassembled lines. This requires an additional source code server with source codes, but once you have it, itContinue reading “pycrashext – A rich python extension”

Python/CRASH API aka pkydump

I am dealing with vmcore analysis for the most of my daily work. To speed up the analysis, I needed some extra command set on top of the commands ‘crash’ is providing. Luckily there is a tool names ‘pkydump’ which is a crash extension and also provides a way to implement extensions using python. IContinue reading “Python/CRASH API aka pkydump”

Normal user process shows “root:root” permission in /proc//

If you are seeing ‘root:root’ under /proc// for your process, it means the process is in Zombie state. #include int main() { pid_t pid; int i; for (i = 0; ; i++) { pid = fork(); if (pid > 0) { break; } else { exit(0); } } while (1) { sleep(1); } return 0;Continue reading “Normal user process shows “root:root” permission in /proc//”

How to disassemble a module from a vmcore

There are times that you have to deal with a module which you don’t have source code. Only thing we can do is disassemble it, but if you don’t have actual module binary, this is also tough. Luckily, vmcore has all the code loaded into the memory. So, here’s the steps to get disassembled codeContinue reading “How to disassemble a module from a vmcore”

How dump trace is generated in Linux kernel

Note for myself to remember how call trace is generated in Kernel /* * The architecture-independent dump_stack generator */ void dump_stack(void) { unsigned long stack; printk(“Pid: %d, comm: %.20s %s %s %.*sn”, current->pid, current->comm, print_tainted(), init_utsname()->release, (int)strcspn(init_utsname()->version, ” “), init_utsname()->version); show_trace(NULL, NULL, &stack); } void show_trace(struct task_struct *task, struct pt_regs *regs, unsigned long *stack) {Continue reading “How dump trace is generated in Linux kernel”

How to find out who is killing my process

There’s a time a process is suddenly killed, but has no idea which process or who killed it. There are couple of ways to identify, but using SystemTap is one clear way to identify it. SystemTap is a script language that can be loaded into Linux kernel and interact safely in kernel to monitoring orContinue reading “How to find out who is killing my process”

What’s virtual address limit of 32bit/64bit Linux kernel?

RHEL 5 code 32bit: include/asm-i386/processor.h /* * User space process size: 3GB (default). */ #define TASK_SIZE (PAGE_OFFSET) 64bit: include/asm-x86_64/processor.h /* * User space process size. 47bits minus one guard page. */ #define TASK_SIZE64 (0x800000000000UL – 4096) /* This decides where the kernel will search for a free chunk of vm * space during mmap’s. */Continue reading “What’s virtual address limit of 32bit/64bit Linux kernel?”

Quotes from “The Secret Life Of Walter Mitty”

To see the world, things dangerous to come to, to see behind walls, draw closer, to find each other, and to feel. That is the purpose of life. Sometimes I don’t. If I like a moment, for me, personally, I don’t like to have the distraction of the camera. I just want to stay inContinue reading “Quotes from “The Secret Life Of Walter Mitty””

Personal memo for ‘Automatic NUMA Balancing’

Automatic NUMA Balancing It is described in Documentation/sysctl/kernel.txt numa_balancing Enables/disables automatic page fault based NUMA memory balancing. Memory is moved automatically to nodes that access it often. Enables/disables automatic NUMA memory balancing. On NUMA machines, there is a performance penalty if remote memory is accessed by a CPU. When this feature is enabled the kernelContinue reading “Personal memo for ‘Automatic NUMA Balancing’”