Using logwarn to manage logs along with Nagios

I needed to check on a reject file and notify Nagios right away. The reject file rotate everyday with the following format: "C"yyyymmdd.rej (C20120209.rej). The file looks like this:


The objectives were the following:
- Check file every 1 minute from 9:30 am - 4:00 pm
- The first line contains "ACMEFixLogs" and it is not a reject payload
- Notify the number of rejects and the name of the file
- If after the notification, there are no rejects, do not notify

The simplest way is to leverage some type of log checkers that reads the content and saves the last line in a temporary file. This way, the next check will start from the saved line and the content will not be repeated. I found logwarn and I really like it. Mostly because it is has a Nagios plugin and this is my notification engine. Also, logwarn has a regex filter that I could use for my file.

Here is how I solve the problem:
- Create a shell script as a Nagios plugin (exit 0 if everything is "OK", exit 2 if there are any errors.
- Leverage the capabilities of logwarn to handle the reading of the file
- Use the regex so that I disregard the first line "ACMEFixLogs" using the logwarn "!"
- Use Linux's word count to find out how many rejected lines there are
- If there are any reject payloads notify (exit 2)
- If there are none, notify that everything is fine (exit 0)

Here is the script:

Executing a command line via Groovy to Nagios

The other day I got stuck trying to execute a Linux command via Groovy. The command needed to pass several parameters to Nagios (my monitoring system). When I was trying to pass all the parameter via String, this did not work. The way that I was able to resolve it is using a string array. Here is an example:

Automation, monitor with Nagios, and passive checks

The first thing that we did during my previous company (a start-up) was to automate everything. Granted, there are some stuff that you cannot automate for whatever reasons, but you should try to automate as much as possible because of scalability. Once an automated process is setup, then the only thing pending is to make sure if it's running fine and if there are any warnings, if it is not running, etc. Nagios is a great tool for monitoring these type of things. There is almost an infinite amount of things that you can do with it. Previously, it was a pain to install, but lately, the installation is very straight forward.

In Nagios, there are two types of checks: active checks and passive checks. Active checks is a constant check - very much like heartbeat in high-availability architectures. The example that I always tell managers is a constant "are you OK?" conversation between the Nagios server and its clients. This is great if you need to have consistent check on a particular process. For example, uptime, drive space, CPU load, memory usage are processes that should be monitored every "n" time (every 5 minutes). For those process that are asynchronous and should be triggered by a particular event, then we use passive checks. In my case I had the following requirements:

• Client(s) can provide me a file at anytime between 9 am - 6:30 pm
• The file should contain a specific format
• If the format is not valid, we need to contact the client
• If the format is correct, persist it into the DB
• Once it is in the DB, then we launch another process and performed some statistical calculations

Passive checks can be for a host or a particular service. In this example, I will covered the steps to configure a service.

1. Install Nagios
2. Configure a particular service for this component
3. Create an external application that checks the state of the application (in my case I used Groovy and shell script - groovysh)
4. Write to an external command file

This is the picture of the process for Nagios:


There are a few configurations that needs to be enabled to have the passive checks work in the nagios configuration (/usr/local/nagios/etc/nagios.cfg). Make sure that the followings are set to "1" (enable):
- accept_passive_service_checks=1
- check_external_commands=1

There should also be a "command_file" with some type of path. For example: command_file=/usr/local/nagios/var/rw/nagios.cmd

Then, configure the service check and enable the passive_checks_enabled. This will be done in the host configuration file (localhost in my case):
vi /usr/local/nagios/etc/objects/localhost.cfg



Restart Nagios:
sudo -i service nagios restart

You should be able to see the "asynch_client_files" service in the localhost. The next step would be to check if the passive check is working via the command file. The way that Nagios knows about any passive events is by writing into a file (nagios.cmd). The following parameters are needed:

where...

timestamp is the time in time_t format (seconds since the UNIX epoch) that the service check was perfomed (or submitted). Please note the single space after the right bracket.
host_name is the short name of the host associated with the service in the service definition
svc_description is the description of the service as specified in the service definition
return_code is the return code of the check (0=OK, 1=WARNING, 2=CRITICAL, 3=UNKNOWN)
plugin_output is the text output of the service check (i.e. the plugin output)

Temporarly, you can get the timestamp by doing the following command in linux: date +%s

To execute the test, we do the following in the terminal screen:
In this case, I had to login as root:

If you go back to the Nagios site: http://localhost/nagios you should be able to see that the new change has been done and now the service shows a status of "OK" with the comment "The security master is up to date"

The only thing missing is configuring an application that sets the status of the page. You can use the language of your choice and use crontab or something else to execute the application.

Unix Admin Mantra - "Only the paranoid survives"

I am not a Unix admin. I had to pick it up due to the fact that I'm working on a small start-up company. I quickly learned that the mantra of Andrew Grove from Intell, "Only the paranoid survives" best fit for this type of job. System admins need to be ahead of the curve. The other day, I went to restart the only windowns server that I have and noticed an error in one of my Unix servers. The message was that I had a bad memory chip. How can I check if everything is OK, specially since my server are in a data center? Then I found out from Linux Journal that I can use SNMP and Nagios to get this type of monitoring. I will be playing with it along as with Ruby for the next couple of weeks. I hope to get status of memory modules, fans, and power supplies in each of my servers.

SNMP (Simple Network Management Protocol) is a network protocol designed for monitoring network-attached devices. It uses OIDs (Object IDentifiers) for defining the information,

known as MIBs (Management Information Base), that can be monitored. The design is extensible, so vendors can define their own items to be monitored

Tomcat Event Handler - privileges

In the previous post regarding Tomcat and Event handler, there was one major problems, privileges. NRPE is a daemon that run in the background when it's launched by nagios. The event handler developed launched "kill" when tomcat did not stop gracefully. Also, when trying to execute the application, the following error happened:

sudo: sorry, you must have a tty to run sudo

The solution is having a service application running the main application. In this example, restart-tomcat-eventhandler.sh is the service which calls restart-tomcat.sh. Also, I made the applicaiton (restart-tomcat.sh) to run in the background mode. But first, below are the changes that we need for the sudoers (visudo), alter the default for requiretty and the privileges for the nagios user to:

Defaults:nagios    !requiretty
...
nagios  ALL=(ALL) NOPASSWD:/opt/tomcat/bin/catalina.sh,/bin/kill,/opt/tomcat/bin/startup.sh

The service application also takes care for the logging mechanism, and verifies that only one process is running for the restart-tomcat.sh

#!/bin/sh
#
# Application that launches the restarting of tomcat.
# The application will be launch in the background but
# its logging will be set in the LOGGER
#
LOGGER=/usr/local/nagios/libexec/eventhandlers/restart-tomcat.log
EVENT_HANDLER_APP=/usr/local/nagios/libexec/eventhandlers/restart-tomcat.sh
echo "Restarting Tomcat `date`...."
count=` ps -ef | grep -c '[r]estart-tomcat.sh' `
echo "Total process running: $count"
typeset -i count
if [ $count -ge 1 ]
then
echo "Another process is running and so the script will stop `date`"
exit
fi
$EVENT_HANDLER_APP >> $LOGGER  2>&1 &

The actual event handler is the following:

#!/bin/bash

#
# tomcat-restart.sh - tomcat restart script for cron
# Need to have access to the sudo to restart the tomcat
# Also, modify the visudo

echo "---------------------`date`---------------------"
CATALINA_PATH=/opt/apache-tomcat-6.0.18
CATALINA_SCRIPT=catalina.sh

echo "CATALINA_HOME : $CATALINA_PATH"

# Verify that tomcat is not running.  If it is, stop it gracefully
# get the tomcat pid
tomcat_pid=`ps -ef | grep java | grep tomcat | cut -c10-14`
echo "Tomcat PID is: $tomcat_pid"

if [ -n "$tomcat_pid" ]
then
echo "Stopping tomcat ..."
sudo $CATALINA_PATH/bin/$CATALINA_SCRIPT stop
# give tomcat 60 seconds to shutdown gracefully
sleep 60
fi

tomcat_pid=`ps -ef | grep java | grep tomcat | cut -c10-14`
# if tomcat_pid exists, kill the process
if [ -n "$tomcat_pid" ]
then
echo "Noticed that process is still running trying to kill it"
sudo kill $tomcat_pid
sleep 60
fi

tomcat_pid=`ps -ef | grep java | grep tomcat | cut -c10-14`
# if tomcat_pid still exists, really kill the process
if [ -n "$tomcat_pid" ]
then
echo "Forcefully killing the process for tomcat $tomcat_pid..."
sudo kill -n 9 $tomcat_pid
sleep 60
fi

# restart tomcat
echo "`date` Starting tomcat..."
sudo $CATALINA_PATH/bin/$CATALINA_SCRIPT start
echo "`date` Finished starting tomcat"
echo "---------------------------------------------"

Nagios and Tomcat Event Handler

The first thing that we need to make sure is understand how Nagios work. Assuming that Tomcat is in a remote server, then there is a "nagios" user, and this needs to have rights to restart tomcat (CATALINA_HOME/bin/catalina.sh stop). If you try to stop tomcat, the nagios user will get the following error:

su nagios
/usr/local/tomcat-18version/bin/catalina.sh stop
Jun 15, 2009 2:54:18 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:
java.io.FileNotFoundException: /opt/apache-tomcat-6.0.18/conf/server.xml (Permission denied)

The best thing to do is to create a group "tomcat", provide privileges on CATALINA_HOME to this group, and add the user "nagios" to this group. In this case, the user download Tomcat in the following directory: /opt/apache-tomcat-6.0.18/. Use the "root" user to do the following steps:
I created a symbolic link so I don't have to change anything in case Tomcat is upgraded.

ln -s /opt/apache-tomcat-6.0.18/ /opt/tomcat

Now, if you do something like this:

ls -l /opt
tomcat -> /opt/apache-tomcat-6.0.18/

Create a group using the groupadd command and add the "nagios" user to this group:

groupadd tomcat

Add the existing nagios user to the tomcat group.

usermod -g tomcat nagios

Add privileges to the /opt/tomcat to the group "tomcat" and the original . First check the id for the user

[root@dev opt]# id nagios
uid=501(nagios) gid=503(tomcat) groups=503(tomcat)
chgrp -R tomcat apache-tomcat-6.0.18
chgrp -R tomcat tomcat

#To test that the nagios user is able to restart run the following command:
su nagios
/usr/local/tomcat-18version/bin/catalina.sh stop

Privileges also need to be provided to restart the tomcat server and killed in case the tomcat doesn't shutdown. Since only root can start certain ports (i.e. port 80), edit the sudoers file (visudo):

##add the following line below "root    ALL=(ALL)       ALL"
nagios  ALL=(ALL) NOPASSWD:/opt/tomcat/bin/catalina.sh,/bin/kill

Now, add the event handler. Create a file in /user/local/nagios/libexec/eventhandler/restart-tomcat.sh

#!/bin/bash

#
# tomcat-restart.sh - tomcat restart script for cron
#
echo "`date`------------ Shutting down tomcat---------------"
CATALINA_PATH=
CATALINA_SCRIPT=catalina.sh

# Verify that tomcat is not running.  If it is, stop it gracefully
# get the tomcat pid
tomcat_pid=`ps -ef | grep java | grep tomcat | cut -c10-14`
echo "Tomcat PID is: $tomcat_pid"

if [ -n "$tomcat_pid" ]
then
echo "Stopping tomcat ..."
sudo $CATALINA_PATH/bin/$CATALINA_SCRIPT stop
# give tomcat 60 seconds to shutdown gracefully
sleep 60
fi

tomcat_pid=`ps -ef | grep java | grep tomcat | cut -c10-14`
# if tomcat_pid exists, kill the process
if [ -n "$tomcat_pid" ]
then
echo "Noticed that process is still running trying to kill it"
sudo kill $tomcat_pid
sleep 60
fi

tomcat_pid=`ps -ef | grep java | grep tomcat | cut -c10-14`
# if tomcat_pid still exists, really kill the process
if [ -n "$tomcat_pid" ]
then
echo "Forcefully killing the process for tomcat $tomcat_pid..."
sudo kill -n 9 $tomcat_pid
sleep 60
fi

# restart tomcat
echo "`date` Starting tomcat..."
sudo $CATALINA_PATH/bin/$CATALINA_SCRIPT start
echo "`date` Finished starting tomcat"

Configure an application that runs the event-handler.sh (restart-tomcat-eventhandler.sh). This way when the application restart, a log that monitors everything:

#!/bin/sh

echo "Restarting Tomcat `date`" >> /usr/local/nagios/libexec/eventhandlers/restart-tomcat.log
/usr/local/nagios/libexec/eventhandlers/restart-tomcat.sh >> /usr/local/nagios/libexec/eventhandlers/restart-tomcat.log
echo "Finished `date`" >> /usr/local/nagios/libexec/eventhandlers/restart-tomcat.log
echo "-------------------------Finished `date`-----------------------------"

In the Nagios server

Create the event handler: /opt/user/local/nagios/event-handler/restart-tomcat-eventhandler.sh

#!/bin/sh
#
# Event handler script for restarting the web server on the local machine
#
# Note: This script will only restart the web server if the service is
#       retried 3 times (in a "soft" state) or if the web service somehow
#       manages to fall into a "hard" error state.
#


# What state is the HTTP service in?

case "$1" in
OK)
  # The service just came back up, so don't do anything...
  ;;
WARNING)
  # We don't really care about warning states, since the service is probably still running...
  ;;
UNKNOWN)
  # We don't know what might be causing an unknown error, so don't do anything...
  ;;
CRITICAL)
  # Aha!  The HTTP service appears to have a problem - perhaps we should restart the server...

  # Is this a "soft" or a "hard" state?
  case "$2" in
    
  # We're in a "soft" state, meaning that Nagios is in the middle of retrying the
  # check before it turns into a "hard" state and contacts get notified...
  SOFT)
            
          # What check attempt are we on?  We don't want to restart the web server on the first
          # check, because it may just be a fluke!
          case "$3" in
                    
          # Attempt number
          3)
                  echo -n "Hard-> Restarting JBoss..."
                  echo -n "/usr/local/nagios/libexec/check_nrpe -H " $4 " -c restart_jboss"
    
                  /usr/local/nagios/libexec/check_nrpe -H $4 -c restart_jboss

                  ;;
                  esac
          ;;
                    
  # The HTTP service somehow managed to turn into a hard error without getting fixed.
  # It should have been restarted by the code above, but for some reason it didn't.
  # Let's give it one last try, shall we?
  # Note: Contacts have already been notified of a problem with the service at this
  # point (unless you disabled notifications for this service)
  HARD)
          echo -n "Hard-> Restarting Tomcat..."
          echo -n "/usr/local/nagios/libexec/check_nrpe -H " $4 " -c restart-tomcat"
    
          /usr/local/nagios/libexec/check_nrpe -H $4 -c restart-tomcat


          ;;
  esac
  ;;
esac
:

Finally, add these event handler as a command by editing the /usr/local/nagios/etc/nrpe.cfg:

command[restart-tomcat]=/usr/local/nagios/libexec/eventhandlers/restart-tomcat-eventhandler.sh

Test that the command is working correctly by executing the following command from the Nagios server:

Now, add the service to restart the server:

/usr/local/nagios/libexec/check_nrpe -H tomcatserver -c restart-tomcat -t 30

define service{
use generic-service
host_name midc
service_description check_midc_login_page
process_perf_data 1
check_command check_http!-H midc.up-mobile.com -u /midc/doLogin.do -w 5 -c 10
event_handler restart-tomcat
}

Now restart nagios (service nagios restart) and you should be ready.