Python/CRASH API aka pkydump

I am dealing with vmcore analysis for the most of my daily work. To speed up the analysis, I needed some extra command set on top of the commands ‘crash’ is providing.

Luckily there is a tool names ‘pkydump’ which is a crash extension and also provides a way to implement extensions using python. I developed some my own extensions using this API. Before I discuss my own set, I would like to share how you can using ‘pykdump’. You can vendor provided ‘crash’, but in that case, you may not able to use ‘pkydump’ due to version mismatch. By having your own build, you can avoid that situation.

pykdump

Pykdump is python framework to access Linux dumps programmatically from ‘crash’ and tools written using it
- Python/CRASH API

Building from GIT

Make a working directory


$ mkdir mycrash
$ cd mycrash

Download pykdump source code


$ git clone git://git.code.sf.net/p/pykdump/code pykdump

Download ‘Python source from https://www.python.org/


$ wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tar.xz
$ tar xvf Python-3.6.5.tar.xz
$ cd Python-3.6.5
$ ./configure CFLAGS='-fPIC'
$ cp /root/mycrash/pykdump/Extension/Setup.local-3.6 Modules/Setup.local
$ make

Download ‘crash’ source from http://people.redhat.com/anderson/crash-7.2.3.tar.gz


$ cd ~
$ wget http://people.redhat.com/anderson/crash-7.2.3.tar.gz
$ tar xvf crash-7.2.3.tar.gz  
$ cd crash-7.2.3
$ 
$ make

Compile pykdump source code


$ cd ..
$ cd pykdump/Extension/
$ ./configure -c ~/mycrash/crash-7.2.3 -p ~/mycrash/Python-3.6.5
$ make
$ make install
$ echo "extend /root/.crash.d/mpykdump.so" > ~/.crashrc

Start crash and check if the extension is loaded


crash> extend
SHARED OBJECT                  COMMANDS
/cores/crashext/mpykdump64.so  epython xportshow crashinfo taskinfo nfsshow hanginfo fregs per_cpu

Useful commands provided by PyKdump

crashinfo : 1st-pass analysis which is also provides lots of convenient data


crash> crashinfo -h
Usage: crashinfo [options]

Options:
  -h, --help            show this help message and exit
  -v                    verbose output
  -q                    quiet mode - print warnings only
  --fast                Fast mode - do not run potentially slow tests
  --sysctl              Print sysctl info.
  --ext3                Print EXT3 info.
  --blkreq              Print Block I/O requests
  --blkdevs             Print Block Devices Info
  --scsi                Print SCSI Dvices Info
  --filelock            Print filelock info.
  --stacksummary        Print stacks (bt) categorized summary.
  --findstacks=FINDSTACKS
                        Print stacks (bt) containing functions that match the
                        provided pattern
  --checkstacks          Check stacks of all threads for corruption
  --decodesyscalls=DECODESYSCALLS
                        Decode Syscalls on the Stack
  --keventd_wq          Decode keventd_wq
  --kblockd_wq          Decode kblockd_workqueue
  --lws                 Print Locks Waitqueues and Semaphores
  --devmapper           Print DeviceMapper Tables
  --runq                Print Runqueus
  --semaphore=SEMA      Print 'struct semaphore' info
  --rwsemaphore=RWSEMA  Print 'struct rw_semaphore' info
  --mutex=MUTEX         Print Mutex info
  --umem                Print User-space Memory Usage
  --ls=LS               Emulate 'ls'. You can specify either dentry address or
                        full pathname
  --workqueues          Print Workqueues - just for some kernels
  --version             Print program version and exit

Checking sysctl parameters with ‘crashinfo’


crash> crashinfo --sysctl | grep panic
fs.xfs.panic_mask    0
kernel.hardlockup_panic 1
kernel.hung_task_panic 0
kernel.panic         0
kernel.panic_on_io_nmi 0
kernel.panic_on_oops 1
kernel.panic_on_stackoverflow 0
kernel.panic_on_unrecovered_nmi 0
kernel.panic_on_warn 0
kernel.softlockup_panic 0
kernel.unknown_nmi_panic 0
vm.panic_on_oom      0

xportshow : Networking related information


crash> xportshow -h
usage: xportshow [-h] [-a] [-v] [-r] [--program PROGRAM] [--pid [PID]]
                 [--netfilter] [--softnet] [--summary] [-s] [-i]
                 [--interface IF1] [--decode DECODE [DECODE ...]]
                 [--port PORT] [-l] [-t] [--tcpstate TCPSTATE] [-u] [-w] [-x]
                 [--sysctl] [--devpack] [--arp] [--rtcache]
                 [--skbuffhead SKBUFFHEAD] [--netns NETNS] [--version]
                 [--everything]

optional arguments:
  -h, --help            show this help message and exit
  -a                    print all sockets
  -v                    verbose output
  -r                    Print routing table. Adding -v prints all routing
                        tables and policies
  --program PROGRAM     print sockets for cmdname
  --pid [PID]           print sockets for PID
  --netfilter           Print Netfilter Hooks
  --softnet             Print Softnet Queues
  --summary             Print A Summary
  -s, --statistics      Print Statistics
  -i                    Print Interface Info
  --interface IF1       Limit output to the specified interface only
  --decode DECODE [DECODE ...]
                        Decode iph/th/uh
  --port PORT           Limit output to the specified port (src or dst)
  -l, --listening       Print LISTEN sockets only
  -t                    Print TCP Info
  --tcpstate TCPSTATE   Limit display for this state only, e.g. SYN_SENT
  -u, --udp             Print UDP Info
  -w, --raw             Print RAW Info
  -x, --unix            Print UNIX Info
  --sysctl              Print sysctl info for net.
  --devpack             Print dev_pack info
  --arp                 Print ARP & Neighbouring info
  --rtcache             Print the routing cache
  --skbuffhead SKBUFFHEAD
                        Print sk_buff_head
  --netns NETNS         Set net ns address
  --version             Print program version and exit
  --everything          Run all functions available for regression testing

Sungju's Slow Life