There’s a time you want to check the local variables or other entries in the stack. Below is an example that was crashed in ‘kmem_freepages’ and needed to check why it’s crashed whiling freeing it. PID: 26 TASK: ffff81027f9197a0 CPU: 0 COMMAND: “events/0” #0 [ffff81027f92fa90] crash_kexec at ffffffff800aaa0c #1 [ffff81027f92fb50] __die at ffffffff8006520f #2 [ffff81027f92fb90]… Continue reading

If the vmcore was generated by human and you want to check who actually was, you might need to check the related process. There are various options in ‘ps’ command, so, you would be able to check it with below steps. crash> ps -a 6326 PID: 6326 TASK: ffff810402165820 CPU: 1 COMMAND: “fuser” ARG: fuser… Continue reading

The real merit of the vmcore is that you can trace the code with the current value each variable holds. Here you can find one example that traces the filesystem which ended up with the corrupted data entry somehow. http://pagead2.googlesyndication.com/pagead/show_ads.js crash> bt PID: 6326 TASK: ffff810402165820 CPU: 1 COMMAND: “fuser” #0 [ffff8103b54efa80] crash_kexec at ffffffff800b099c… Continue reading

With ‘bt’ command you can check the processes that occupied the CPU at the time of crash. But, there is the time you want to check other processes’s backtrace to see what interaction had been established between processes. You can check it by using one of below two methods. crash> set 24960 PID: 24960 COMMAND:… Continue reading

There are times the crash had happened because of the lack of memory. Or times that system had hard time because of the memory issue. To check those, you can use below command. crash> kmem -i PAGES TOTAL PERCENTAGE TOTAL MEM 16464824 62.8 GB —- FREE 3807386 14.5 GB 23% of TOTAL MEM USED 12657438… Continue reading

Now we are ready to deep dive into vmcore analysis to confirm what went wrong in the system. In general, first thing I’m checking is the system’s log. crash> log …. …. end_request: I/O error, dev sdajm, sector 0 end_request: I/O error, dev sdajm, sector 8 end_request: I/O error, dev sdajm, sector 0 NMI Watchdog… Continue reading

As I had mentioned in the previous article, you can launch crash analysis by running the below command $ crash vmcore vmlinux The order of the file doesn’t matter as far as the file exist there. But, there’s a time that crash command fails as a vmcore is highly incomplete. In that case, you can… Continue reading