Tag: Linux

Rsyslog to clean up /var/log/messages

Situation: Not all applications and utilities have been developed to log their messages intelligently or to their own location.

Complication: This can lead to logging bloat, especially for messages that are purely informational and noisy. /var/log/messages can be overwhelmed by this and can make it harder to figure out actual problems.

Question: How can you redirect messages to clean up /var/log/messages?

Answer: rsyslog to the rescue!

 

For the purposes of this post I will be analyzing the program amazon-ssm-agent. This is an Amazon proprietary program necessary to run AWS Run Command. This is a good example because it was developed to have it’s own log file, but also still fills /var/log/messages. We will:

  1. Go through the workflow of syslog, systemd and how messages get into logs
  2. Look at messages in /var/log/messages that need to be filtered
  3. Configure rsyslog to send all application logs to it’s own file
  4. Use logrotate to create a good policy for how long logs stay around
  5. Celebrate clean logs

 

ADCLI on CentOS 6

If you are trying to bind a Linux machine to Active Directory, a very simple tool to use is ADCLI. You can use it to join servers, query AD, and also add/delete objects. It is not currently available through the yum repos, however. Here’s how to get past that:

  1. Download http://ftp.redhat.com/pub/redhat/linux/enterprise/6Server/en/os/SRPMS/adcli-0.8.1-1.el6.src.rpm
  2. You’ll need to rebuild the source rpm like this:
    1. rpmbuild –rebuild adcli-0.8.1-1.el6.src.rpm
  3. You may end up with needing other dependencies such as openldap-devel and xmlto (and potentially others). You can install these through yum:
    1. yum install -y openldap-devel xmlto
  4. To bind a new server:
    1. adcli join -U <ADadminUser> domain
    2. Enter password
  5. Verify the bind was a success by querying AD:
    1. adcli info <domain>

You should now be able to login with any AD user on that machine and do application level Active Directory integration

AWS Run Command

Amazon Web Services recently came out with a new feature called “Run Command”. If you have instances in AWS it allows you to send a set of commands to a subset (or all) of your instances, with the ability for extended logging of the output sent to an S3 bucket, if you wish.

People will sometimes use tools such as Puppet to send a new system configuration that may only be a single command, such as a systemctl enable command. But the AWS Run Command will let you do this without having to create a Puppet module for a small set of commands.

Pre-requisites

  • amazon-ssm-agent must be running.
  • Security groups must be configured to allow this agent

Security Groups

AWS Identity and Access Management has a default SSM Policy which you can apply to your instances if you just search for it in IAM:

Screen Shot 2016-04-20 at 12.34.13 AM.png

After you have applied the AmazonEC2RoleforSSM you need to get the amazon-ssm-agent service running. You can bake this into whichever AMI baking tool you are using, or configuration management (Chef, Puppet etc). But to test you can set it up like this for Linux instances upon creation or in a script run by your configuration management software:

cd /tmp 
curl https://amazon-ssm-<your-region>.s3.amazonaws.com/latest/linux_amd64/amazon-ssm-agent.rpm -o amazon-ssm-agent.rpm
yum -y install amazon-ssm-agent.rpm

Start the service for CentOS 7.x using:

sudo systemctl start amazon-ssm-agent

For CentOS 6.x use:

service start amazon-ssm-agent

Once amazon-ssm-agent is running, then you can issue commands to your instances. Here’s the basics on how to do it:

In AWS Console –> EC2 –> Commands

Screen Shot 2016-04-19 at 11.35.52 AM

Then select the type of Command you wish to run, for example a simple Shell Script:

Screen Shot 2016-04-19 at 11.28.30 AM

You can then choose to send the full output of your commands to an S3 bucket:

Screen Shot 2016-04-19 at 11.45.14 AM

Once the command is run you can view the output by clicking “View result”:

Screen Shot 2016-04-19 at 11.42.00 AM

 

The AWS Run Command is extremely powerful for one-off commands to be run without the overhead and delay of a configuration management tool. The amazon-ssm-agent takes a little extra time to set up, but once you have it started you can utilize the Run Command at no cost other than minimal cross-region traffic costs.