Situation: Not all applications and utilities have been developed to log their messages intelligently or to their own location.
Complication: This can lead to logging bloat, especially for messages that are purely informational and noisy. /var/log/messages can be overwhelmed by this and can make it harder to figure out actual problems.
Question: How can you redirect messages to clean up /var/log/messages?
Answer: rsyslog to the rescue!
For the purposes of this post I will be analyzing the program amazon-ssm-agent. This is an Amazon proprietary program necessary to run AWS Run Command. This is a good example because it was developed to have it’s own log file, but also still fills /var/log/messages. We will:
- Go through the workflow of syslog, systemd and how messages get into logs
- Look at messages in /var/log/messages that need to be filtered
- Configure rsyslog to send all application logs to it’s own file
- Use logrotate to create a good policy for how long logs stay around
- Celebrate clean logs