Unix Admin Mantra - "Only the paranoid survives"

I am not a Unix admin. I had to pick it up due to the fact that I'm working on a small start-up company. I quickly learned that the mantra of Andrew Grove from Intell, "Only the paranoid survives" best fit for this type of job. System admins need to be ahead of the curve. The other day, I went to restart the only windowns server that I have and noticed an error in one of my Unix servers. The message was that I had a bad memory chip. How can I check if everything is OK, specially since my server are in a data center? Then I found out from Linux Journal that I can use SNMP and Nagios to get this type of monitoring. I will be playing with it along as with Ruby for the next couple of weeks. I hope to get status of memory modules, fans, and power supplies in each of my servers.

SNMP (Simple Network Management Protocol) is a network protocol designed for monitoring network-attached devices. It uses OIDs (Object IDentifiers) for defining the information,

known as MIBs (Management Information Base), that can be monitored. The design is extensible, so vendors can define their own items to be monitored

Teams - different breeds

I am a big believer of teams. When I completed my MBA, it was a huge shock for me. Most of my work was in teams. Being a Computer Science graduate, I always wanted to do the work either alone or with other individuals that were developers or knowledgeable in the computer field. During my MBA at the University of Miami, I learned much about teams. One thing that stuck with me is that cohesive teams become more united during great circumstances. Think about Hurricane Andrew, Hurricane Katrina, or 9-11. During these times, people came together for one particular cause, their communities. The same thing happens when the team itself feels threaten.

I recently moved, and noticed that the drier of my new home was not working. Yesterday I called my landlord to explain to him about the problem. His voice started to break, and told me that his 17 year-old son drowned the day before and that he was going to take care of the drier issue later. Words cannot explain how I felt. I have a seven-year-old and he and my wife are my life. I cannot imagine what I would do if something similar happened to me. I told my landlord that he shouldn't worry about it, and that I would fix it and for him to focus on his family.

A day later, three gentleman came to the house and fixed the problem with my drier. They said that they came in behalf of my landlord. I found out that my landlord is the chief of the Miami Fire Department and they are proud to help their chief. One of the gentlemen told me,

Fire Departments are a different animal. We take care of our people.

I was deeply moved. It also made me think, would my team do the same thing for me? Would my friends? My wife and I picked up a sympathy card to send it to our landlord. I can't imagine what is he going through, but I would take this opportunity to remind me how vulnerable we are. Indeed, the hardest times reveal the true friendships.

Tomcat Event Handler - privileges

In the previous post regarding Tomcat and Event handler, there was one major problems, privileges. NRPE is a daemon that run in the background when it's launched by nagios. The event handler developed launched "kill" when tomcat did not stop gracefully. Also, when trying to execute the application, the following error happened:

sudo: sorry, you must have a tty to run sudo

The solution is having a service application running the main application. In this example, restart-tomcat-eventhandler.sh is the service which calls restart-tomcat.sh. Also, I made the applicaiton (restart-tomcat.sh) to run in the background mode. But first, below are the changes that we need for the sudoers (visudo), alter the default for requiretty and the privileges for the nagios user to:

Defaults:nagios    !requiretty
...
nagios  ALL=(ALL) NOPASSWD:/opt/tomcat/bin/catalina.sh,/bin/kill,/opt/tomcat/bin/startup.sh

The service application also takes care for the logging mechanism, and verifies that only one process is running for the restart-tomcat.sh

#!/bin/sh
#
# Application that launches the restarting of tomcat.
# The application will be launch in the background but
# its logging will be set in the LOGGER
#
LOGGER=/usr/local/nagios/libexec/eventhandlers/restart-tomcat.log
EVENT_HANDLER_APP=/usr/local/nagios/libexec/eventhandlers/restart-tomcat.sh
echo "Restarting Tomcat `date`...."
count=` ps -ef | grep -c '[r]estart-tomcat.sh' `
echo "Total process running: $count"
typeset -i count
if [ $count -ge 1 ]
then
echo "Another process is running and so the script will stop `date`"
exit
fi
$EVENT_HANDLER_APP >> $LOGGER  2>&1 &

The actual event handler is the following:

#!/bin/bash

#
# tomcat-restart.sh - tomcat restart script for cron
# Need to have access to the sudo to restart the tomcat
# Also, modify the visudo

echo "---------------------`date`---------------------"
CATALINA_PATH=/opt/apache-tomcat-6.0.18
CATALINA_SCRIPT=catalina.sh

echo "CATALINA_HOME : $CATALINA_PATH"

# Verify that tomcat is not running.  If it is, stop it gracefully
# get the tomcat pid
tomcat_pid=`ps -ef | grep java | grep tomcat | cut -c10-14`
echo "Tomcat PID is: $tomcat_pid"

if [ -n "$tomcat_pid" ]
then
echo "Stopping tomcat ..."
sudo $CATALINA_PATH/bin/$CATALINA_SCRIPT stop
# give tomcat 60 seconds to shutdown gracefully
sleep 60
fi

tomcat_pid=`ps -ef | grep java | grep tomcat | cut -c10-14`
# if tomcat_pid exists, kill the process
if [ -n "$tomcat_pid" ]
then
echo "Noticed that process is still running trying to kill it"
sudo kill $tomcat_pid
sleep 60
fi

tomcat_pid=`ps -ef | grep java | grep tomcat | cut -c10-14`
# if tomcat_pid still exists, really kill the process
if [ -n "$tomcat_pid" ]
then
echo "Forcefully killing the process for tomcat $tomcat_pid..."
sudo kill -n 9 $tomcat_pid
sleep 60
fi

# restart tomcat
echo "`date` Starting tomcat..."
sudo $CATALINA_PATH/bin/$CATALINA_SCRIPT start
echo "`date` Finished starting tomcat"
echo "---------------------------------------------"

Starting to learn Ruby on Rails

Started to learn Ruby on Rails. Bought a couple of books (I'm a book addict). I end up purchasing Simply Rails 2 by Patrick Lenz and Advance Rails Recipes by Mike Clark since I need to have a project up-and-running in a couple of weeks.

IT Project Kill Switch

Waldo Moreira pointed me to the article How to Make Profit which brings an important lesson for IT and project decisions. The article is based on a decision made by the CEO of Rakspace Managed Hosting, Graham Weston, when he passed on a $20 million deal with Morgan Stanley. His decision was based on the fact that the deal was not profitable enough. To be more precise, it was 5% less than the original 15% profit margin for Rakspace.

The article explains that many companies lack the discipline of true profit or economic value added,

...Lots of big corporations don't make a true profit. That is equally true of small businesses, which can be so desperate to close deals early on that they neglect to really look at the numbers. As a result, line managers are clueless about the cost of capital and the returns — or the lack thereof — they are generating.

Jim Collins wrote in his master piece Good to Great how leaders "Confront the Brutal Facts". The great leaders had the following patterns: all of the them gather data before making a decission, then make excellent use of it, and finally use it to confront their decisions head-on. This is what Weston end up doing. After analyzing the venture with Morgan Stanley, he noticed that Rakspace was going to make 10% profit, 5% less than the original 15% profit margin. In his new book, How The Mighty Falls, Jim Collins talks about the five steps that companies take before failing. The second step is called "Undisciplined Pursued of More",

...More scale, more growth, more acclaim, more of whatever those in power seem as success...Although complacency and resistance to change remains dangers to any successful enterprise, overreaching better capture how the mighty falls.

There has to be some type of threshold that allows senior management to take the bold step and say "no" to specific projects. Senior management need to look beyond the numbers. In the HBR essay, The Truths About IT Cost, Susan Cramm writes about what drives up IT costs. She identified seven such truths. Perhaps the most interesting is "Project Failures are too High". Being an IT director, I'm faced with different "wish list" of projects from marketing, sales, and senior management. IT should not be the one to define whether or not a project should launch. As Cramm explains,

Managing these truth is tricky. IT can't do it alone, because simply saying no to business partners harms relationships with them.

Senior Manager should provide a threshold, a magic number, like the 15% of Weston and IT should raise the flag when a project is going down the wrong path. As Cramm says,

Establish a "kill switch" rules for projects.

If a project is out of the initial budget and has been modified twice and beta deployment still not occurred, KILL IT!

Nagios and Tomcat Event Handler

The first thing that we need to make sure is understand how Nagios work. Assuming that Tomcat is in a remote server, then there is a "nagios" user, and this needs to have rights to restart tomcat (CATALINA_HOME/bin/catalina.sh stop). If you try to stop tomcat, the nagios user will get the following error:

su nagios
/usr/local/tomcat-18version/bin/catalina.sh stop
Jun 15, 2009 2:54:18 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:
java.io.FileNotFoundException: /opt/apache-tomcat-6.0.18/conf/server.xml (Permission denied)

The best thing to do is to create a group "tomcat", provide privileges on CATALINA_HOME to this group, and add the user "nagios" to this group. In this case, the user download Tomcat in the following directory: /opt/apache-tomcat-6.0.18/. Use the "root" user to do the following steps:
I created a symbolic link so I don't have to change anything in case Tomcat is upgraded.

ln -s /opt/apache-tomcat-6.0.18/ /opt/tomcat

Now, if you do something like this:

ls -l /opt
tomcat -> /opt/apache-tomcat-6.0.18/

Create a group using the groupadd command and add the "nagios" user to this group:

groupadd tomcat

Add the existing nagios user to the tomcat group.

usermod -g tomcat nagios

Add privileges to the /opt/tomcat to the group "tomcat" and the original . First check the id for the user

[root@dev opt]# id nagios
uid=501(nagios) gid=503(tomcat) groups=503(tomcat)
chgrp -R tomcat apache-tomcat-6.0.18
chgrp -R tomcat tomcat

#To test that the nagios user is able to restart run the following command:
su nagios
/usr/local/tomcat-18version/bin/catalina.sh stop

Privileges also need to be provided to restart the tomcat server and killed in case the tomcat doesn't shutdown. Since only root can start certain ports (i.e. port 80), edit the sudoers file (visudo):

##add the following line below "root    ALL=(ALL)       ALL"
nagios  ALL=(ALL) NOPASSWD:/opt/tomcat/bin/catalina.sh,/bin/kill

Now, add the event handler. Create a file in /user/local/nagios/libexec/eventhandler/restart-tomcat.sh

#!/bin/bash

#
# tomcat-restart.sh - tomcat restart script for cron
#
echo "`date`------------ Shutting down tomcat---------------"
CATALINA_PATH=
CATALINA_SCRIPT=catalina.sh

# Verify that tomcat is not running.  If it is, stop it gracefully
# get the tomcat pid
tomcat_pid=`ps -ef | grep java | grep tomcat | cut -c10-14`
echo "Tomcat PID is: $tomcat_pid"

if [ -n "$tomcat_pid" ]
then
echo "Stopping tomcat ..."
sudo $CATALINA_PATH/bin/$CATALINA_SCRIPT stop
# give tomcat 60 seconds to shutdown gracefully
sleep 60
fi

tomcat_pid=`ps -ef | grep java | grep tomcat | cut -c10-14`
# if tomcat_pid exists, kill the process
if [ -n "$tomcat_pid" ]
then
echo "Noticed that process is still running trying to kill it"
sudo kill $tomcat_pid
sleep 60
fi

tomcat_pid=`ps -ef | grep java | grep tomcat | cut -c10-14`
# if tomcat_pid still exists, really kill the process
if [ -n "$tomcat_pid" ]
then
echo "Forcefully killing the process for tomcat $tomcat_pid..."
sudo kill -n 9 $tomcat_pid
sleep 60
fi

# restart tomcat
echo "`date` Starting tomcat..."
sudo $CATALINA_PATH/bin/$CATALINA_SCRIPT start
echo "`date` Finished starting tomcat"

Configure an application that runs the event-handler.sh (restart-tomcat-eventhandler.sh). This way when the application restart, a log that monitors everything:

#!/bin/sh

echo "Restarting Tomcat `date`" >> /usr/local/nagios/libexec/eventhandlers/restart-tomcat.log
/usr/local/nagios/libexec/eventhandlers/restart-tomcat.sh >> /usr/local/nagios/libexec/eventhandlers/restart-tomcat.log
echo "Finished `date`" >> /usr/local/nagios/libexec/eventhandlers/restart-tomcat.log
echo "-------------------------Finished `date`-----------------------------"

In the Nagios server

Create the event handler: /opt/user/local/nagios/event-handler/restart-tomcat-eventhandler.sh

#!/bin/sh
#
# Event handler script for restarting the web server on the local machine
#
# Note: This script will only restart the web server if the service is
#       retried 3 times (in a "soft" state) or if the web service somehow
#       manages to fall into a "hard" error state.
#


# What state is the HTTP service in?

case "$1" in
OK)
  # The service just came back up, so don't do anything...
  ;;
WARNING)
  # We don't really care about warning states, since the service is probably still running...
  ;;
UNKNOWN)
  # We don't know what might be causing an unknown error, so don't do anything...
  ;;
CRITICAL)
  # Aha!  The HTTP service appears to have a problem - perhaps we should restart the server...

  # Is this a "soft" or a "hard" state?
  case "$2" in
    
  # We're in a "soft" state, meaning that Nagios is in the middle of retrying the
  # check before it turns into a "hard" state and contacts get notified...
  SOFT)
            
          # What check attempt are we on?  We don't want to restart the web server on the first
          # check, because it may just be a fluke!
          case "$3" in
                    
          # Attempt number
          3)
                  echo -n "Hard-> Restarting JBoss..."
                  echo -n "/usr/local/nagios/libexec/check_nrpe -H " $4 " -c restart_jboss"
    
                  /usr/local/nagios/libexec/check_nrpe -H $4 -c restart_jboss

                  ;;
                  esac
          ;;
                    
  # The HTTP service somehow managed to turn into a hard error without getting fixed.
  # It should have been restarted by the code above, but for some reason it didn't.
  # Let's give it one last try, shall we?
  # Note: Contacts have already been notified of a problem with the service at this
  # point (unless you disabled notifications for this service)
  HARD)
          echo -n "Hard-> Restarting Tomcat..."
          echo -n "/usr/local/nagios/libexec/check_nrpe -H " $4 " -c restart-tomcat"
    
          /usr/local/nagios/libexec/check_nrpe -H $4 -c restart-tomcat


          ;;
  esac
  ;;
esac
:

Finally, add these event handler as a command by editing the /usr/local/nagios/etc/nrpe.cfg:

command[restart-tomcat]=/usr/local/nagios/libexec/eventhandlers/restart-tomcat-eventhandler.sh

Test that the command is working correctly by executing the following command from the Nagios server:

Now, add the service to restart the server:

/usr/local/nagios/libexec/check_nrpe -H tomcatserver -c restart-tomcat -t 30

define service{
use generic-service
host_name midc
service_description check_midc_login_page
process_perf_data 1
check_command check_http!-H midc.up-mobile.com -u /midc/doLogin.do -w 5 -c 10
event_handler restart-tomcat
}

Now restart nagios (service nagios restart) and you should be ready.

AT&T and Verizon is making a mess out of PSMS

U.S. Telcos have tried over and over again to gain a hold of spammers using MT premium SMS (PSMS), but instead they have made a mess of imposing APIs for aggregators. Verizon has tried OIOO and now AT&T came up with its own version OPPC. Both flows tried to do the same thing, enforce the opt-in to the users so they know what are they being billed. OPPC's flow uses a "start" message that the aggregators have to send to AT&T. This message contains the parameters for AT&T to build the opt-in message (price, description, shortcode, etc). However, what if your campaign is targetting non-english speakers and you want to control the opt-in message? What happens if you are using a trivia or a chat? Should the users receive an opt-in for every question or every chat that they send? Eventually, AT&T came up with a "except-tag". This is the exception to the OPPC rule. I think that eventually, everyone would like to have an "except tag". How about the spammer, wouldn't they eventually use this except tag? This is just frustrating. I understand the use of these API, but why do we need to impose different API for different operators? Can we all just get a standard? Perhaps we need to think about doing the billing via MO like Europe and Latin America is doing.