This tutorial is about the OOM killer and how it can impact our Linux operating system.
OOM killer is a shortcut to Out of Memory Killer and is a Linux component designed to prevent RAM exhaustion. The OOM Killer has one task, and that task is to check if there is enough available memory, and if there is not, then select a process to kill to free up some RAM. The situations with exhausted memory occur because the Linux kernel has over-allocated too much for its processes. In the following paragraphs, we will talk about memory over-allocation, the lifecycle of the OOM killer, how it selects which process should be killed, the OOM killer messages, etc.
Table of Contents
Over Allocation of Memory
Every process on the system requests a block of memory from the kernel. The kernel allocates memory more than the process needs when it is starting. The kernel is aware of this scenario. However, it over-allocates the RAM. This means that if the system has 4 GB of RAM, the kernel might allocate 5GB for all running processes which is 1 GB more than the limit. Normally, at the beginning, all processes do not use the memory allocated in their blocks, but after some time, they may use all the memory. There is a moment when the system does not have enough physically available RAM and is not able to serve all processes.
The OOM Killer invoking
In addition to the previous paragraph, we know that at this moment there is not enough RAM on the server. The next is the OOM killer to get invoked. It reviews all the running processes, but before selecting which process will be killed, there are a couple of steps that need to be verified:
If there is enough swap space, then we are not out of memory (OOM).
If there are more than 5 seconds since the last failure, we are not out of memory (OOM).
Finally, if there are not 10 failures in the last 5 seconds, then we are not out of memory (OOM).
If one of these conditions is the opposite, then we are out of memory, and the OOM killer is selecting which process should be killed. The selection is based on the select_bad_process() function. It decides by stepping through each running process and making a calculation of how suitable it is for killing the chosen process based on the function badness():
badness_for_task = total_vm_for_task / (sqrt(cpu_time_in_seconds) * sqrt(sqrt(cpu_time_in_minutes)))
The formula above can be explained in simple words: The badness task function selects the process that uses a lot of memory but has not been running for a long time. Long-running processes are most unlikely to cause memory shortage. Once this function selects the process, the OOM killer kills it immediately.
OOM Killer Log
When the website stops working, most probably some service stopped running. We need to find out which service that is, and what process may have been killed. To do that, we can execute the following command:
cat /var/log/messages | grep "Killed"
As an output, we can get a message similar to this:
Mar 02 14:04:48 host kernel: Out of Memory: Killed process 2574 (mysqld).
This indicates that the OOM killer killed the MySQL process due to insufficient memory. That is why the MySQL service stopped working and caused our website to experience downtime. Killing the MySQL service often irreversibly corrupts the database system, and we may lose all data we store. There are a couple of steps that we may take to avoid these scenarios of killing the MySQL process by overcommitting memory. Keep in mind that this is not recommended. We need to simply upgrade the server so we can have more available resources.
That’s it. We explained a little bit more about the OOM killer and how it affects the server. Of course, you do not have to take care of that when your server is out of memory. Simply sign up for one of our NVMe VPS plans and submit a support ticket. Our admins will help you with any aspect of memory consumption on your server. Feel free to contact us. We are available 24/7.
PS. If you liked this post about the OOM killer on Linux, please share it with your friends on social networks or simply leave a comment in the comments section. Thank you.