Sungju's Slow Life

When there is high usage in kernel side, most common reason is high usage in SLAB. As the SLAB memory blocks are all data, it is hard to tell which parts of code was consuming this memory blocks.

Fortunately, there is this kernel option slub_debug=U which is saving backtrace of the allocation and freeing calls. This requires additional space, so, it is mainly used for the debugging purposes only.

Once you boot with slub_debug=U, all the allocations are recorded in each slab blocks with the data structure struct track.

To make this information available, the system needs to have the below option in red colour.

Once the system is booted with this option, you can collect a vmcore and run crash with that. Other option is just run crash on live system. Either way, you can see the current slab by run kmem -s.

To see the actual slabs in kmem_cache, you can run the below.

…

It shows each SLABs allocated in the kmem_cache with actual usage (ALLOCATED) as well as total allocation (TOTAL) in that slab.

To see the actual objects in a slab, you can specify slab address in -S option.

Allocated object will be displayed with [ and ]. In the above, ffff9fa5cad5c2c0 and ffff9fa5cad5c6e0 are the only free entry.

In this kmalloc-32, each block uses 352 bytes even though the actual object size is 32 bytes. After the object (data) area, the struct track follows.

With track, you can find the allocation function as well as full cal trace. It also shows which pid called this function, but the related function already might gone.

So, by doing this step for the all the objects, you can find what is the major cause of this allocation which will be very tough and tedious operation.

So, to minimise that, I wrote a script that you can find in my pycrashext extension. Using the below command will gives you the summary of the allocation pattern.

With this, we easily can check what was the cause. This is even works when Z option is used which is filling Red Zone at the beginning and the middle.