Sungju's Slow Life

Personal journal


  • How to track SLAB usage using slub_debug=U

    When there is high usage in kernel side, most common reason is high usage in SLAB. As the SLAB memory blocks are all data, it is hard to tell which parts of code was consuming this memory blocks. Fortunately, there is this kernel option slub_debug=U which is saving backtrace of the allocation and freeing calls.… Continue reading

  • Difference between ‘pgscank/s’ and ‘pgscand/s’

    In the sar output like below, there are pgscank and pgscand. In the man page, it says. This description is not much helpful to understand the difference between those two columns. These data are from kernel code which is showing in vmstat. It is basically updated when there’s memory reclaiming. If the reclaiming was happened… Continue reading

  • 5-level paging in KVM

    Latest kernels such as RHEL8 and above are supporting 5-level page table. You can check if this is available/enabled by looking at /boot/config-$(uname -r). If you don’t want to use 5-level page table, you can disable that by adding ‘no5lvl’ as a kernel parameter in grub file (/boot/grub2/grubenv) and reboot the system. This 5 level… Continue reading

  • Install mpykdump on local machine

    Here is the sequence of commands to build mpykdump on your system. This is based on crash version 8.0.0. You may want to check what version of crash your system has to match it. Below is the sequence on RHEL8. You can ignore the error from ‘crash’ compile as we are not going to use… Continue reading

  • How to check page caches in the kernel memory.

    If you would like to know what files were occupying the page caches which you can see with ‘free’ in command line or ‘kmem -I’ in kernel memory, you can start with ‘/proc/sys/vm/drop_caches’. I wrote a script that is extracting the page caches from the vmcore based on the above information. You can find the… Continue reading

  • How intel_idle works

    When the system is in IDLE state which means nothing to run and swapper is running, it calls cpuidle_idle_call() like shown in the below. This cpuidle_idle_call() is called from arch_cpu_idle(). cpuidle_idle_call() is the main idle loop which is checking idle driver and do further steps if the driver is installed and active. It can change… Continue reading

  • SystemTap to monitor SLAB usage

    Checking memory leak in SLAB is not an easy task. It is happening in kernel side and it can go through several different route. Here I want to show how we can use SystemTap to monitor those SLAB alloc/free activities. As a first step you need to find out the monitoring points. SLAB can be… Continue reading

  • How to write mpykdump extension

    If you are dealing with a vmcore (Linux memory dump), you must be familiar with ‘crash’. It is a powerful tool, but it doesn’t cover all the data you can find in Linux kernel. So, there comes ‘mpykdump’ which is a crash extension which understands python code. mpykdump comes with many prebuilt commands that you… Continue reading

  • Where’s PageSlab() macro???

    If you are having hard time to find the definition of PageSlab() in linux kernel, here’s the answer. In include/linux/page-flags.h: Above becomes like below during compile time. Continue reading

  • What is ‘page_cache’, how it is managed and how ‘drop_caches’ dropping this pages?

    – The “buffers/cache” values reported by free include the page cache, but not the dentry cache which is saved in slab ‘dentry_cache’. – page cache is increased and decreased based on the disk access activities and managed by each super block (it means each disk). – ‘echo 1 > /proc/sys/vm/drop_caches’ frees page caches by calling… Continue reading

  • How to calculate available memory for mem_cgroup.

    mem_cgroup is checking available amount of memory the group can charge by calling the below function. /** * mem_cgroup_margin – calculate chargeable space of a memory cgroup * @memcg: the memory cgroup * * Returns the maximum amount of memory @mem can be charged with, in * pages. */ static unsigned long mem_cgroup_margin(struct mem_cgroup *memcg)… Continue reading

  • What happens if you try two commands ‘ethtool -p ‘ and ‘ethtool ‘ in parallel.

    If you start ‘ethtool -p ‘ and also start ‘ethtool ‘ after that, you may see the delays in ‘ethtool ‘ command. It is because any ethtool commands start by taking ‘rtnl_lock’ and ‘ethtool -p’ is keep running for LED on/off. In the below, bnx2x’s identity function just turns on or off the led. get_settings()… Continue reading