Month: July 2016

Rsyslog to clean up /var/log/messages

Situation: Not all applications and utilities have been developed to log their messages intelligently or to their own location.

Complication: This can lead to logging bloat, especially for messages that are purely informational and noisy. /var/log/messages can be overwhelmed by this and can make it harder to figure out actual problems.

Question: How can you redirect messages to clean up /var/log/messages?

Answer: rsyslog to the rescue!

 

For the purposes of this post I will be analyzing the program amazon-ssm-agent. This is an Amazon proprietary program necessary to run AWS Run Command. This is a good example because it was developed to have it’s own log file, but also still fills /var/log/messages. We will:

  1. Go through the workflow of syslog, systemd and how messages get into logs
  2. Look at messages in /var/log/messages that need to be filtered
  3. Configure rsyslog to send all application logs to it’s own file
  4. Use logrotate to create a good policy for how long logs stay around
  5. Celebrate clean logs

 

Linux tools to parse through files

Situation: You have a lot of results in a file that you need to move through and single out and add or subtract to fields, compare entries, or many other needs

Complication: There are a TON of Linux tools to do this

Question: What can do what?

Answer: I will add to this post with specific examples consistently

 

Example 1: Search through a file list and return only the filename, not the full path

Let’s say you want to just look for missing files in a non-similar directory structure between multiple file lists. One way is to look through each file without the file path and get just the file name. Let’s say filelist1 has this list:

/tmp/derp/directoryone/file1
/tmp/derp/directoryone/file2
/tmp/derp/directoryone/file3
/tmp/derp/directoryone/file4

Let’s now say filelist2 has this list:

/tmp/der/directorytwo/file1
/tmp/der/directorytwo/file2
/tmp/der/directorytwo/file3

In this case you want to just figure out that you’re missing file 4, so that you could sync just that file and not all the contents of directoryone. So to do this you could use cat to show the file contents, and pipe it to cut to get just the filename

less filelist1 | cut -d/ -f4

The -d/ option is saying to use the / as the delimiter, and the -f4 is saying to return the 4th entry

This would return

file1
file2
file3
file4

You could then send it to a new file that had just the filenames with >>

less filelist1 | cut -d/ -f4 >> filelist1_clean
less filelist2 | cut -d/ -f4 >> filelist2_clean

You can now easily find just the files missing by running

diff filelist1_clean filelist2_clean

You would then get

file4

Now you can make your sync off of this