How to Deal with Huge (and Growing) Log Files in Linux

If you’ve managed a Linux server for any length of time, you’re familiar with the problem of log files. They can sometimes be difficult enough to even find in the first place, and then you’re sometimes confronted with a file that’s hundreds of MB in size (or even GB). Searching through it is a pain, and they can eventually even start eating up your storage space.

This tutorial will show you how to deal with this problem. We’re going to make use of the inbuilt Linux package for system logs that’s already in place. This means you don’t need to install any new packages! All you need to know is how to add the configuration for your own log files.

Seeing How Linux Logs are Rotated with logrotate

There are a bunch of system log files in the following location:

/var/log/

In the screenshot below, I list all the files in that directory and show the file size in MB:

As you can see, there are lots of large log files that are datestamped. These are generated by the “logrotate” by default. Looking at the timestamps, we can see that they’re generated once every seven days. So for each log file here, logrotate does the following:

1. Renames the log file with a timestamp
2. Creates a new empty log file with the same name as before

Checking Logrotate’s Default Configuration

Logrotate checks the following file for its configuration:

/etc/logrotate.conf

In this, we can see that it gets all the important information about:

1. How often it should rotate the logs
2. How many backups should it keep
3. What kind of suffix it should add to the old logs
4. Whether or not it should compress them
5. etc…

Here’s a screenshot of the logrotate configuration file:

By default, the old log files are not compressed. You can change this by simply uncommenting the “compress” directive as shown above.

Adding Files to logrotate

How does logrotate know which logs to work on? The files in this directory:

/etc/logrotate.d

For example, in our “secure” log that we saw in the first screenshot, we can figure out which file in logrotate.d contains it by using a simple grep command, as detailed in our guide on grep for advanced users:

grep -r secure /etc/logrotate.d

The screenshot below reveals which file is responsible for the “secure” log:

Opening “syslog”, we can see that it tells logrotate to work on a number of files:

All of these are located in /var/log/. So each file contains:

1. The names of the log files that it should rotate
2. Directives specific to that particular file
3. In the absence of directives, the defaults from logrotate.conf will be used

Creating your Own Logrotate File for your Logs

We can follow the template above to create an entry with logrotate that tells us it to process our own files in the same way.

Get the Owner and Group for your Directory

To make sure that all permissions are in place, we need logrotate to run with the appropriate permissions. Navigate to the directory that contains the log file(s) you want to process and get the owner as well as the group using this command:

ls -l -d

For example, the /var/log directory has the owner as” root”, and the group as “root” as well:

Create a Logrotate File with the Configuration

Next, create a new file in /etc/logrotate.d/ and paste the following sample configuration into it:

/var/log/newlogfile {

size 10M

compress

delaycompress

su root root

}

Replace /var/log/newlogfile with the location of your own log file. Also, replace su root root with the owner and group name you got in the previous step. Here’s a list of logrotate directives you can use. The “delaycompress” directive tells logrotate to only compress files that are 2 versions old, or older. This way you have quick and easy access to the most recently compressed log file.

You can also use the following lines within the curly brackets block to run scripts after and before your log files are rotated:

postrotate
// scripts for post rotation go here
endscript

As you can see, Linux has a complete solution for log file rotation. All you need to do is add a few lines of code to the existing framework, and your own log files can be rotated without any hassles!

If you are one of our managed VPS hosting clients, you can simply ask our system administrators to deal with huge files on your Linux server. They are available 24/7 and will take care of your request immediately.

If you find this blog post useful, please share it with your friends via social media networks, or if you have any question please leave a comment below and we will reply to it. Thanks!