Sungju's Slow Life

If you have a vmcore needs to be analysed, you should have to download related kernel-debuginfo package unless your kernel does have all the debug information in the live kernel.

You might easily check the kernel version of the vmcore by run ‘uname -r’, but safest way would be checking vmcore itself.

You can use one of the below methods to check the kernel version.

$ strings vmcore | head
KDUMP   
Linux
(none)
2.6.18-194.el5
#1 SMP Tue Mar 16 21:52:39 EDT 2010
x86_64
(none)
 @R 
0 (@

or

$ strings vmcore | fgrep -m1 'Linux '

or

$ crash --osrelease vmcore

http://pagead2.googlesyndication.com/pagead/show_ads.js

You can download kernel-debuginfo from your linux distributor. In my case, I can download it from Red Hat customer portal (‘https://rhn.redhat.com/rhn/software/downloads/SupportedISOs.do’). After downloading related package you just need to extract it to your current directory. You don’t need to install it in your system just to check the vmcore. For the RPM file, you can use below command.

$ rpm2cpio kernel-debuginfo-2.6.18-194.el5.x86_64.rpm | cpio -idv
...
...
./usr/lib/debug/usr/src/kernels/2.6.18-194.el5-x86_64/scripts/kallsyms.debug
./usr/lib/debug/usr/src/kernels/2.6.18-194.el5-x86_64/scripts/kconfig
./usr/lib/debug/usr/src/kernels/2.6.18-194.el5-x86_64/scripts/kconfig/conf.debug
./usr/lib/debug/usr/src/kernels/2.6.18-194.el5-x86_64/scripts/mod
./usr/lib/debug/usr/src/kernels/2.6.18-194.el5-x86_64/scripts/mod/mk_elfconfig.debug
./usr/lib/debug/usr/src/kernels/2.6.18-194.el5-x86_64/scripts/mod/modpost.debug
./usr/lib/debug/usr/src/kernels/2.6.18-194.el5-x86_64/scripts/pnmtologo.debug
1194352 blocks

Now you can launch ‘crash’ to start analysing the vmcore.

$ crash vmcore usr/lib/debug/lib/modules/2.6.18-194.el5/vmlinux 

crash 6.0.9
Copyright (C) 2002-2012  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

      KERNEL: usr/lib/debug/lib/modules/2.6.18-194.el5/vmlinux
    DUMPFILE: vmcore  [PARTIAL DUMP]
        CPUS: 8
        DATE: Thu Sep  6 17:37:33 2012
      UPTIME: 04:37:20
LOAD AVERAGE: 38.31, 14.07, 7.76
       TASKS: 2034
    NODENAME: mytestnode
     RELEASE: 2.6.18-194.el5
     VERSION: #1 SMP Tue Mar 16 21:52:39 EDT 2010
     MACHINE: x86_64  (2813 Mhz)
      MEMORY: 63.1 GB
       PANIC: ""
         PID: 5543
     COMMAND: "ocfs2dc"
        TASK: ffff81087e14b0c0  [THREAD_INFO: ffff810c72776000]
         CPU: 2
       STATE: TASK_RUNNING (NMI)

crash>