Reverse
-
Print callgraph of a function
Sometimes you may want to see what functions are called in a function in multiple level. Below command in my extension may help. crash> edis -c irq_exit {irq_exit} -+- {rcu_irq_exit} -+- {warn_slowpath_null} |- {idle_cpu} |- {tick_nohz_stop_sched_tick} -+- {ktime_get} | |- {update_ts_time_stats} | |- {sched_clock_idle_sleep_event} | |- {rcu_needs_cpu} | |- {select_nohz_load_balancer} | |- {rcu_enter_nohz} | |-… Continue reading
-
Why error message not goes into pipe nor redirected path in ‘crash’?
In the below example, the error always shows in the console. crash> sym ffffffffa02ef86 > /dev/null sym: invalid address: ffffffffa02ef86 This ‘sym’ command is implemented in ‘void cmd_sym(void)’ function in crash. /* * This command may be used to: * * 1. Translate a symbol to its value. * 2. Translate a value to it… Continue reading
-
What happens if numa=off is provided in kernel parameter?
If “numa=off” is in kernel boot parameter, it will mark ‘numa_off’ global variable which will be checked during initialization function which is ‘x86_numa_init()’ in x86_64. This will make it not call ‘numa_init’ if numa_off is 1. static __init int numa_setup(char *opt) { if (!opt) return -EINVAL; if (!strncmp(opt, “off”, 3)) numa_off = 1; #ifdef CONFIG_NUMA_EMU… Continue reading
-
An example case with some of my commands
System got high load average and it wasn’t responding for long which is a typical hang situation. It shows total 56 tasks in D (Uninterruptible) state and 5 tasks were in D state longer than 120 seconds which is considered as a hung task. Let’s see what it was waiting for in this process. Alright,… Continue reading
-
‘edis’ command now shows stack values
One of the repeating job is finding the argument values passed from the caller. Now by just run ‘edis’, you can see some stack values after the operation code as a comment in yellow color. Continue reading
-
Clocksource tsc unstable
The reason of the below message 2013-07-16T05:00:05.181538-04:00 xxxxxx kernel: Clocksource tsc unstable (delta = -95170507948 ns). Enable clocksource failover by adding clocksource_failover kernel parameter. clocksource structure /** * struct clocksource – hardware abstraction for a free running counter * Provides mostly state-free accessors to the underlying hardware. * This is the structure used for system… Continue reading
-
pycrashext – A rich python extension
Based on Pykdump, I wrote a set of plugins named ‘pycrashext’ which is basically trying to help to reduce the troubleshooting time. My favorite command in this set is ‘edis’ which can display source code in between disassembled lines. This requires an additional source code server with source codes, but once you have it, it… Continue reading
-
Python/CRASH API aka pkydump
I am dealing with vmcore analysis for the most of my daily work. To speed up the analysis, I needed some extra command set on top of the commands ‘crash’ is providing. Luckily there is a tool names ‘pkydump’ which is a crash extension and also provides a way to implement extensions using python. I… Continue reading
-
What’s TAINT_WARN?
TAINT_WARN is explained in kernel/panic.c as ‘Taint on warning’. static const struct tnt tnts[] = { … { TAINT_WARN, ‘W’, ‘ ‘ }, } /** … * ‘W’ – Taint on warning. … */ This flag is turned on from “__WARN()” to confirm that the system had ‘WARNING’ messages once or more time. #define __WARN()… Continue reading
-
How to disassemble a module from a vmcore
There are times that you have to deal with a module which you don’t have source code. Only thing we can do is disassemble it, but if you don’t have actual module binary, this is also tough. Luckily, vmcore has all the code loaded into the memory. So, here’s the steps to get disassembled code… Continue reading
-
How dump trace is generated in Linux kernel
Note for myself to remember how call trace is generated in Kernel /* * The architecture-independent dump_stack generator */ void dump_stack(void) { unsigned long stack; printk(“Pid: %d, comm: %.20s %s %s %.*sn”, current->pid, current->comm, print_tainted(), init_utsname()->release, (int)strcspn(init_utsname()->version, ” “), init_utsname()->version); show_trace(NULL, NULL, &stack); } void show_trace(struct task_struct *task, struct pt_regs *regs, unsigned long *stack) {… Continue reading
-
How to find out who is killing my process
There’s a time a process is suddenly killed, but has no idea which process or who killed it. There are couple of ways to identify, but using SystemTap is one clear way to identify it. SystemTap is a script language that can be loaded into Linux kernel and interact safely in kernel to monitoring or… Continue reading
About Me
A software engineer who loves any technologies that makes life easier. That’s why I love Linux and Mac at the same time.
Sungju's Slow Life 