Logrotate: The Complete Guide

Sébastien Puyet

Sébastien Puyet

— 8 min read
Logrotate: The Complete Guide

Log files are essential for troubleshooting and monitoring, but they can quickly consume disk space and degrade system performance if left unmanaged. A single busy web server can generate gigabytes of logs daily, potentially filling up disk partitions and causing service outages.

Logrotate is a Linux system utility that automatically manages log files by rotating, compressing, and removing old logs based on configurable rules. It runs as a scheduled cron job and ensures your systems maintain optimal performance while preserving important log data for the appropriate retention period.

What is Logrotate and Why You Need It

Logrotate solves the fundamental problem of unbounded log growth. Without proper log management, applications and services will continue writing to log files indefinitely, eventually consuming all available disk space. This leads to system crashes, failed deployments, and potential data loss.

The utility works by periodically "rotating" log files - moving the current log to a timestamped archive file and creating a fresh log file for new entries. It can compress old logs to save space, automatically delete logs older than a specified age, and coordinate with running services to ensure continuous logging during rotation.

Key benefits include:

  • Disk space management: Prevents log files from consuming all available storage
  • Performance optimization: Keeps active log files small for faster I/O operations
  • Compliance support: Maintains logs for required retention periods while automatically purging old data
  • Operational reliability: Reduces risk of disk-full incidents that can crash services

However, logrotate isn't always the right solution. In containerized environments where logs are streamed to stdout/stderr, or when using centralized logging systems like ELK stack or Splunk, log rotation may be handled by other components. Modern cloud-native applications often rely on log aggregation services rather than local file rotation.

Quick Start: Get Logrotate Working in 5 Minutes

Most Linux distributions include logrotate by default. Verify your installation and check the version:

logrotate --version

If logrotate isn't installed, install it using your package manager:

# Ubuntu/Debian
sudo apt update && sudo apt install logrotate

# RHEL/CentOS/Fedora
sudo yum install logrotate

Let's create a simple configuration for a custom application log. Create a new file in the logrotate configuration directory:

sudo nano /etc/logrotate.d/myapp

Add this basic configuration:

/var/log/myapp/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0644 www-data www-data
}

This configuration tells logrotate to:

  • daily - Rotate logs once per day
  • rotate 7 - Keep 7 days of archived logs
  • compress - Compress old log files with gzip
  • delaycompress - Wait one rotation cycle before compressing
  • missingok - Don't error if log files are missing
  • notifempty - Don't rotate empty files
  • create 0644 www-data www-data - Create new log files with specific permissions and ownership

Test your configuration without actually rotating files:

sudo logrotate -vdf /etc/logrotate.d/myapp

The flags mean: -v (verbose output), -d (debug mode, don't actually rotate), -f (force rotation even if not needed). You should see detailed output showing what logrotate would do.

To perform an actual test rotation:

sudo logrotate -vf /etc/logrotate.d/myapp

Understanding Logrotate Configuration Structure

Logrotate uses a hierarchical configuration system with global defaults and service-specific overrides. The main configuration file is /etc/logrotate.conf, which sets system-wide defaults and includes service-specific configurations from /etc/logrotate.d/.

The main configuration file typically looks like this:

# Global options
weekly
rotate 4
create
dateext
compress

# Include service-specific configurations
include /etc/logrotate.d

# System-specific logs
/var/log/wtmp {
    monthly
    create 0664 root utmp
    rotate 1
}

Never edit the main configuration file directly. Instead, create individual files in /etc/logrotate.d/ for each service or application. This approach provides several advantages:

  • Package management: Software packages can install their own logrotate configurations
  • Maintainability: Changes are isolated and easier to track
  • Override capability: Service-specific settings override global defaults

Configuration files use a simple syntax where log file paths are followed by configuration blocks in curly braces. You can specify multiple files or use wildcards:

# Single file
/var/log/myapp.log { ... }

# Multiple specific files
/var/log/app1.log /var/log/app2.log { ... }

# Wildcard pattern
/var/log/myapp/*.log { ... }

Settings in service-specific files override global defaults. For example, if the global configuration sets weekly rotation but your service configuration specifies daily, daily rotation takes precedence for that service.

Essential Logrotate Directives and Options

Understanding key directives helps you build effective log rotation strategies. Here are the most important options:

Rotation Triggers

  • daily, weekly, monthly - Time-based rotation intervals
  • size 100M - Rotate when file reaches specified size
  • minsize 10M - Rotate based on time, but only if file exceeds minimum size
  • maxsize 500M - Force rotation if file exceeds maximum size, regardless of time

File Management

  • rotate 30 - Number of old log files to keep
  • compress - Compress rotated files with gzip
  • delaycompress - Compress files on the next rotation cycle
  • create 0644 user group - Create new log file with specified permissions
  • copytruncate - Copy file contents then truncate original (for locked files)

Error Handling

  • missingok - Continue if log files don't exist
  • notifempty - Don't rotate empty files
  • ifempty - Rotate even if files are empty (opposite of notifempty)

The choice between create and copytruncate is crucial. Use create when applications can handle log file recreation (most modern applications). Use copytruncate for applications that keep log files open and can't handle file rotation, though this method has a small risk of losing log entries during the copy operation.

Script Integration

You can execute scripts before and after rotation:

/var/log/nginx/*.log {
    daily
    rotate 30
    compress
    delaycompress
    prerotate
        if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
            run-parts /etc/logrotate.d/httpd-prerotate; \
        fi
    endscript
    postrotate
        systemctl reload nginx
    endscript
}

Step-by-Step Configuration Examples

Web Server Logs (Nginx)

Web servers generate high-volume access and error logs. Here's a complete Nginx configuration:

/var/log/nginx/*.log {
    daily
    rotate 52
    compress
    delaycompress
    missingok
    notifempty
    create 0644 www-data adm
    sharedscripts
    postrotate
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 `cat /var/run/nginx.pid`
        fi
    endscript
}

The sharedscripts directive ensures the postrotate script runs only once, even when multiple log files are rotated. The postrotate script sends a USR1 signal to Nginx, instructing it to reopen log files.

Application Logs with Service Restart

Some applications require a full restart to recognize new log files:

/var/log/myapp/application.log {
    weekly
    rotate 12
    compress
    delaycompress
    missingok
    notifempty
    create 0640 myapp myapp
    postrotate
        systemctl restart myapp
    endscript
}

High-Volume Database Logs

Database logs can grow rapidly and may need size-based rotation:

/var/log/mysql/mysql.log /var/log/mysql/error.log {
    size 100M
    rotate 10
    compress
    delaycompress
    missingok
    notifempty
    create 0640 mysql mysql
    postrotate
        mysqladmin flush-logs
    endscript
}

Advanced Logrotate Configuration

Container Environments

In Docker environments, logs are often mounted as volumes. Configure rotation for the host system:

/var/lib/docker/containers/*/*-json.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

Use copytruncate since Docker containers maintain open file handles to their log files.

High-Frequency Rotation

For extremely high-volume logs, you might need hourly rotation. Create a custom cron job:

# Add to /etc/cron.hourly/logrotate-hourly
#!/bin/bash
/usr/sbin/logrotate /etc/logrotate.d/high-volume-app

Configure the logrotate file for hourly execution:

/var/log/high-volume-app/*.log {
    size 1G
    rotate 24
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

Network Filesystems

When logs are stored on NFS or other network filesystems, add the shred option and be cautious with file locking:

/mnt/nfs/logs/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
    shred
}

Security and Permissions Best Practices

Proper security configuration prevents unauthorized access to sensitive log data and ensures logrotate can perform its operations.

File Permissions

Always specify appropriate permissions in the create directive:

# Restrictive permissions for sensitive logs
/var/log/auth.log {
    create 0600 root root
}

# Web server logs accessible to web group
/var/log/apache2/*.log {
    create 0644 root adm
}

SELinux Considerations

On SELinux-enabled systems, logrotate may encounter permission denials. Check for SELinux alerts:

sudo ausearch -m AVC -ts recent | grep logrotate

Common solutions include setting appropriate SELinux contexts or creating custom policies for non-standard log locations.

User Privilege Management

Use the su directive when logrotate needs to run with different privileges:

/var/log/myapp/*.log {
    su myapp myapp
    daily
    rotate 7
    compress
    create 0644 myapp myapp
}

Testing and Validation Techniques

Thorough testing prevents production issues and ensures your rotation strategy works as expected.

Command-Line Testing

Use these flag combinations for different testing scenarios:

# Test configuration syntax without rotation
sudo logrotate -d /etc/logrotate.conf

# Verbose debug output for specific config
sudo logrotate -vdf /etc/logrotate.d/myapp

# Force actual rotation for testing
sudo logrotate -vf /etc/logrotate.d/myapp

Status File Monitoring

Logrotate maintains a status file tracking when each log was last rotated:

sudo cat /var/lib/logrotate/status

This file helps troubleshoot rotation timing issues and verify that rotation is occurring as expected.

Validation Workflow

  1. Test configuration syntax with -d flag
  2. Create test log files with sample content
  3. Force rotation with -f flag
  4. Verify rotated files exist with correct permissions
  5. Check that services restart properly (if using postrotate scripts)
  6. Monitor for several rotation cycles to ensure consistency

Common Issues

Rotation Not Happening

Check if logrotate is running via cron:

# Verify cron service is running
sudo systemctl status cron

# Check for logrotate cron job
ls -la /etc/cron.daily/logrotate

# Review cron logs
sudo grep logrotate /var/log/syslog

Permission Problems

Common permission issues include:

# Check file ownership and permissions
ls -la /var/log/myapp/

# Verify logrotate can write to log directory
sudo -u root touch /var/log/myapp/test-file

# Review SELinux denials (if applicable)
sudo sealert -a /var/log/audit/audit.log

Service Restart Failures

Debug postrotate script issues:

# Test postrotate commands manually
sudo systemctl reload nginx

# Add error handling to postrotate scripts
postrotate
    systemctl reload nginx || logger "Failed to reload nginx after logrotate"
endscript

Disk Space Issues

When rotation fails due to insufficient disk space:

# Check available space
df -h /var/log

# Manually remove old compressed logs
find /var/log -name "*.gz" -mtime +30 -delete

# Adjust rotation frequency or retention period

Monitoring and Maintaining Logrotate

Proactive monitoring ensures logrotate continues working reliably in production environments.

Key Metrics to Monitor

  • Rotation success rate: Track failed rotations via syslog
  • Disk usage trends: Monitor log directory sizes
  • Rotation timing: Verify rotations occur at expected intervals
  • Service restart success: Monitor postrotate script execution

Automated Monitoring Setup

Create a simple monitoring script:

#!/bin/bash
# Check for logrotate errors in the last 24 hours
if grep -q "logrotate.*error" /var/log/syslog; then
    echo "Logrotate errors detected"
    grep "logrotate.*error" /var/log/syslog | tail -5
    exit 1
fi

# Verify status file is recent
if [ $(find /var/lib/logrotate/status -mtime +2) ]; then
    echo "Logrotate status file is stale"
    exit 1
fi

echo "Logrotate monitoring: OK"

Integration with monitoring systems like fivenines.io allows you to track log rotation health alongside other system metrics, providing comprehensive observability for your infrastructure.

Performance Optimization and Scaling

Large log files and high-volume environments require optimization to prevent logrotate from impacting system performance.

I/O Optimization

For large files, consider these strategies:

  • Use copytruncate judiciously: It requires reading the entire file, which can be slow
  • Schedule rotation during low-activity periods: Move logrotate cron jobs to off-peak hours
  • Implement size-based rotation: Prevent files from becoming too large

Compression Strategies

Balance compression ratio against CPU usage:

# Use faster compression for high-volume logs
/var/log/high-volume/*.log {
    compress
    compresscmd /bin/gzip
    compressoptions "-1"  # Fastest compression
    compressext .gz
}

Parallel Processing

For environments with many log files, consider splitting configurations to run in parallel:

# Create separate cron jobs for different log types
# /etc/cron.daily/logrotate-web
# /etc/cron.daily/logrotate-apps
# /etc/cron.daily/logrotate-system

Integration with Modern Log Management

Modern infrastructure often combines traditional log rotation with centralized logging and cloud-based solutions.

Log Shipper Coordination

When using tools like Filebeat or Fluentd, ensure they can handle log rotation:

/var/log/app/*.log {
    daily
    rotate 7
    compress
    delaycompress
    postrotate
        # Signal log shipper to reopen files
        systemctl reload filebeat
    endscript
}

Cloud Integration

Consider archiving old logs to cloud storage:

/var/log/archive/*.log {
    monthly
    rotate 1
    compress
    postrotate
        # Archive to S3 before deletion
        aws s3 cp /var/log/archive/*.gz s3://log-archive-bucket/
        rm -f /var/log/archive/*.gz
    endscript
}

Hybrid Architectures

In hybrid environments, maintain local rotation for immediate troubleshooting while shipping logs to centralized systems for long-term analysis. This approach provides both real-time local access and comprehensive historical analysis capabilities.

With proper configuration and monitoring, logrotate becomes an invisible but essential component of your infrastructure, ensuring systems remain stable and performant while maintaining the log data you need for troubleshooting and compliance.

Read more