Sungju's Slow Life

As I had mentioned in the previous article, you can launch crash analysis by running the below command

$ crash vmcore vmlinux

The order of the file doesn’t matter as far as the file exist there. But, there’s a time that crash command fails as a vmcore is highly incomplete. In that case, you can try below command which at least provide ‘log’ of the system.

$ crash vmcore vmlinux --minimal

Now you are ready to start vmcore analysis. Here are some tips commonly used to check basic information.

Checking the kernel command line

crash> p saved_command_line
saved_command_line = $1 = 0xffffffff804659a0 "ro root=LABEL=/1 console=tty0"

Checking the kernel version and build information

"RHEL"
crash> p linux_banner
linux_banner = $3 = 0xc0624000 "Linux version 2.6.18-238.9.1.el5PAE (mockbuild@x86-004.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-50)) #1 SMP Fri Mar 18 12:50:00 EDT 2011n"
...
 
"CentOS"
crash> p linux_banner
linux_banner = $2 = "Linux version 2.6.18-308.13.1.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-52)) #1 SMP Tue Aug 21 17:10:18 EDT 2012n 00 00 00 00 00 00 00 00"

Some sysctl parameters

crash> p sysctl_panic_on_oom
kernel.panic_on_oom

crash> p panic_on_oops
kernel.panic_on_oops

crash> p sysctl_hung_task_panic
kernel.hung_task_panic

crash> p sysctl_hung_task_check_count
kernel.hung_task_check_count

crash> p sysctl_hung_task_timeout_secs
hung_task_timeout_secs

crash> p watchdog_enabled
kernel.watchdog

crash> p softlockup_thresh
kernel.watchdog_thresh

crash> p softlockup_panic
kernel.softlockup_panic

crash> p unknown_nmi_panic
kernel.unknown_nmi_panic

crash> p panic_on_unrecovered_nmi
kernel.panic_on_unrecovered_nmi