It is one thing to create logs; it is quite another to cope with them. A log file grows without bound unless action is taken and this can cause problems.
Problems with growing log files
Larger files are harder to manipulate.
File systems run out of space.
The information you log may constitute personal data.
A solution to this generic problem of log file growth is log rotation. This involves the regular (nightly or weekly, typically) moving of an existing log file to some other file name and starting fresh with an empty log file. After a period the old log files get thrown away.
Because this is a general issue, many Linux distributions (SLES included) include a general solution that can be applied to any set of log files, not just the web server's. There is an Apache-specific solution (which is provided by the rotatelogs command) but we will use SLES's generic solution, provided in the logrotate package.
Once each night the logrotate program reads in its configuration files telling it which logs to rotate and how to do it. One of these files tells it to rotate Apache's log files.
The main configuration file sets up the defaults and then reads in a directory of instructions for specific sets of log files from the /etc/logrotate.d directory.
# see "man logrotate" for details # rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # uncomment this if you want your log files compressed #compress # uncomment these to switch compression to bzip2 compresscmd /usr/bin/bzip2 uncompresscmd /usr/bin/bunzip2 # former versions had to have the compresscommand set accordingly #compressext .bz2 # RPM packages drop log rotation information into this directory include /etc/logrotate.d # no packages own wtmp -- we'll rotate them here #/var/log/wtmp { # monthly # create 0664 root utmp # rotate 1 #} # system-specific logs may be also be configured here.
/etc/logrotate.conf: commands
Each file should be rotated weekly. The log rotation job runs nightly, though, so this can be changed to daily for a specific log file if desired.
The three commands that specify how often rotation should take place are daily, weekly and monthly.
Keep four sets of log files. The comment is slightly inaccurate; four weeks' worth of logs will be kept if rotation is done weekly. If rotation is done daily then this command means that four days' worth of logs are kept.
After moving the main log file from logfile to logfile.1 a new, empty logfile should be created.
This command instructs the log rotation program to read in every file in this directory. One of these files will correspond to the web server's log files.
The /etc/logrotate.d/apache2 file (part of the apache2 package, not the logrotate package), contains the instructions specific to the web server logs.
/var/log/apache2/access_log { compress dateext maxage 365 rotate 99 size=+4096k notifempty missingok create 644 root root postrotate /etc/init.d/apache2 reload endscript } /var/log/apache2/error_log { compress dateext maxage 365 rotate 99 size=+1024k notifempty missingok create 644 root root postrotate /etc/init.d/apache2 reload endscript }
/etc/logrotate.d/apache2: commands
This specifies that the commands within the curly brackets are to be applied to the access log. A similar block applies to the error log.
Rotated log files should be compressed. Logfiles, which can be very large, typically compress very well.
Archive old versions of log files adding a date extension like YYYYMMDD to the filename.
Remove rotated logs older than 365 days. The age is only checked if a logfile is to be rotated.
Delete old log files once they have been rotated 99 times.
Rotate the file if it becomes bigger than 4MB, even if it was last rotated less than a week ago.
This command instructs the system not to rotate the logs if the current main log file is empty. See the discussion below about whether this is a good idea or not.
This is the instruction not to return an error if a particular log file is not present.
After rotation, create a new logfile owned by user root and group root and with read/write permissions for it's owner and read permissions for everyone else.
Following the rotation of a log file the commands between postrotate and endscript are run. This rotation program runs as root so care must be taken with the commands that appear here.
This instructs Apache to do a "Graceful Restart" in which the master web server demon advises its children to exit after their current request and then re-reads its configuration files and re-opens its log files.
There are two points which must be made about log rotation and changing its settings. These are to do with the presence of editor backup files and the Data Protection Act (1998).
Backup files. The command include /etc/logrotate.d will read in almost every file in that directory. So if you edit the file apache2 and leave behind both apache2 and apache2.old then both these files will be included and your log files will have the log rotation process applied twice. Now, because of the weekly (or monthly or daily) commands the rotation shouldn't actually happen but it is still not certain that the right file will be applied.
Data Protection Act (1998) |
The contents of the log files may constitute personal data. If, along with any information stored within the University, they identify individuals then they definitely do. As a result, you must have a privacy policy, and stick to it. For this reason, the author recommends very strongly that you remove the notifempty command. If your policy says that personal data is only kept for 28 days and it is kept for a week longer because one week's log files happened to be empty then you are in breach of your privacy policy. |