Automation, monitor with Nagios, and passive checks

The first thing that we did during my previous company (a start-up) was to automate everything. Granted, there are some stuff that you cannot automate for whatever reasons, but you should try to automate as much as possible because of scalability. Once an automated process is setup, then the only thing pending is to make sure if it's running fine and if there are any warnings, if it is not running, etc. Nagios is a great tool for monitoring these type of things. There is almost an infinite amount of things that you can do with it. Previously, it was a pain to install, but lately, the installation is very straight forward.

In Nagios, there are two types of checks: active checks and passive checks. Active checks is a constant check - very much like heartbeat in high-availability architectures. The example that I always tell managers is a constant "are you OK?" conversation between the Nagios server and its clients. This is great if you need to have consistent check on a particular process. For example, uptime, drive space, CPU load, memory usage are processes that should be monitored every "n" time (every 5 minutes). For those process that are asynchronous and should be triggered by a particular event, then we use passive checks. In my case I had the following requirements:

• Client(s) can provide me a file at anytime between 9 am - 6:30 pm
• The file should contain a specific format
• If the format is not valid, we need to contact the client
• If the format is correct, persist it into the DB
• Once it is in the DB, then we launch another process and performed some statistical calculations

Passive checks can be for a host or a particular service. In this example, I will covered the steps to configure a service.

1. Install Nagios
2. Configure a particular service for this component
3. Create an external application that checks the state of the application (in my case I used Groovy and shell script - groovysh)
4. Write to an external command file

This is the picture of the process for Nagios:


There are a few configurations that needs to be enabled to have the passive checks work in the nagios configuration (/usr/local/nagios/etc/nagios.cfg). Make sure that the followings are set to "1" (enable):
- accept_passive_service_checks=1
- check_external_commands=1

There should also be a "command_file" with some type of path. For example: command_file=/usr/local/nagios/var/rw/nagios.cmd

Then, configure the service check and enable the passive_checks_enabled. This will be done in the host configuration file (localhost in my case):
vi /usr/local/nagios/etc/objects/localhost.cfg



Restart Nagios:
sudo -i service nagios restart

You should be able to see the "asynch_client_files" service in the localhost. The next step would be to check if the passive check is working via the command file. The way that Nagios knows about any passive events is by writing into a file (nagios.cmd). The following parameters are needed:

where...

timestamp is the time in time_t format (seconds since the UNIX epoch) that the service check was perfomed (or submitted). Please note the single space after the right bracket.
host_name is the short name of the host associated with the service in the service definition
svc_description is the description of the service as specified in the service definition
return_code is the return code of the check (0=OK, 1=WARNING, 2=CRITICAL, 3=UNKNOWN)
plugin_output is the text output of the service check (i.e. the plugin output)

Temporarly, you can get the timestamp by doing the following command in linux: date +%s

To execute the test, we do the following in the terminal screen:
In this case, I had to login as root:

If you go back to the Nagios site: http://localhost/nagios you should be able to see that the new change has been done and now the service shows a status of "OK" with the comment "The security master is up to date"

The only thing missing is configuring an application that sets the status of the page. You can use the language of your choice and use crontab or something else to execute the application.

Getting your Groovy on! - Groovy 1.8

I started working on the new version of Groovy 1.8. Here is a brief description of how you can start using Groovy and the new features of Groovy 1.8. Although, this post has some basic information about Groovy, it is not a "getting started" guideline. If you are new to Groovy, I highly recommend you to go to the following sites. They cover the language in great detail and provide some good documentation:

- Bazhidar Batsov has a great post on Groovy
- Groovy has a great Getting Started Guideline

Installation and Getting Started

- Download the binary distribution and unpack it into some folder (i.e. /groovy-1.8.0)
- (Optional) Create a symbolic link to the groovy directory and I name it "/groovy".
- Set your GROOVY_HOME environment

if you do a "ls -l /groovy" you will get the following:

In my application I do the following (MacBook Pro using TextMate):

To make sure that you have the latest installation, do the following:


Most people I know work with Eclipse. Groovy has a plugin for Eclipse and a tutorial to upgrade the Groovy version inside your IDE.

Using Arrays/List:


SQL and Groovy:


Which will return the following:
-- Marcelo --
-- Antonio --
-- Jorge Luis --
-- Nestor --

In the new Groovy 1.8, there is a feature for pagination:


Where the first two states the starting point and the next two states the max record to fetch.
Output:
XTO (20040809)
DE (20040809)

Tests are perfect for Groovy. You can use the latest JUnit to run all the unit test cases as do in Java. Below I will be using some of the example using JUnit test:

Closures and Arrays/List:
Different people use Groovy in different ways, but everyone agrees that the collection API for Groovy is very powerful. Below are more examples of using List or Arrays along with closures:

As you can see, the each is a closure in Groovy and it iterates through the list. The output is the following:
3
4
5
6
7
11

Closure composition
Closure composition is to be able to combine closure with one another. In some cases you want to combine two type of functions. For example:


Trampoline
Groovy used to have error when using closures in recursive algorithms. They were able to resolve this issue with trampoline. Here is an example of using the factorial function.


Memoize Cache
Memoize is a very interesting tool for Groovy. From the Groovy's documentation:

The return values for a given set of Closure parameter values are kept in a cache, for those memoized Closures. That way, if you have an expensive computation to make that takes seconds, you can put the return value in cache, so that the next execution with the same parameter will return the same result – again, we assume results of an invocation are the same given the same set of parameter values.
There are three forms of memoize functions:
- the standard memoize() which caches all the invocations
- memoizeAtMost(max) call which caches a maximum number of invocation
- memoizeAtLeast(min) call which keeps at least a certain number of invocation results
- and memoizeBetween(min, max) which keeps a range results (between a minimum and a maximum)

Here is an example:

output is:
Fri Jun 24 13:29:15 EDT 2011
Fri Jun 24 13:29:16 EDT 2011
Fri Jun 24 13:29:16 EDT 2011
Fri Jun 24 13:29:17 EDT 2011

Maps and Default Value:
Maps now can have a default value (similar to Ruby). In the case below, if we needed to count the frequency of words in a sentence, we can start a map and store the words and the counter. By configuring the default counter, all the words will have a 0


Groove on brother!

Error svn: Can't create directory in Subversion SVN

I created a respository in svn by doing the following:
svnadmin create clientconnector

But when I tried to connect and create the infrastructure (branches, tags, trunk), I got the following error:


I knew that I need to check the permissions of my connections. Now, I'm using Apache to handle all my repository permissions.

This can be found in the following section:
/etc/httpd/conf.d/subversion.conf



Where all my user's information is stored in the /etc/svn-auth-conf file.

When I checked the directory for my repositories: /data/svn I noticed the following:


The problem is the group. By doing the following I was able to solved the problem with the privileges.

chown -R apache.apache clientconnector

You shouldn't have to restart your apache server, but in case that this doesn't work, restart apache by doing: service httpd restart

When to refactor your code

Like many of my peers, I've worked in different shops and in different domains. First of all, I try NOT to be "that guy" that wants to change everything over night. The number one rule of any developer after finding a good mentor is to become a domain expert. Then, you can start writing value added code. However, there have been times that I need to refactor a piece of core code but the manager or team leader completely denies the need to do the task. Usually, it happens when the managers are either not programmers or the project has a very tight deadline. I usually suggest that it needs to be done since the code is just keep getting "smellier" and follow up with the value added principles of refactoring code: (easy to read, test, DRY, scalable, etc). However, I always leave it up to them to make the decision. This type of attitude is extremely frustrating, specially if you are a full-time employee and if the majority of the other developers want to refactor the project.

The other day I was listening to Standford's Entrepreneurial Thought Leaders. If you haven't listen to them, I highly recommend it. One of my favorite interviews was with David Heinemeier Hansson from 37 Signals and one of the founders of Ruby on Rails. In his speech, he basically said that working for a bad boss was a good thing. Because you learn how NOT to do things.

In one of my early projects, I worked with some six developers. A senior manager, demanded that the project would be done in six months. This was before Scrum or any Agile methodology, or even DDD or TDD was in place. We took the approach of divide and conquered. We separated between teams of threes to work on back-ends, front-end, third-party applications, etc. After six months we tried to integrate the project with all our code. As you can imagine, the project was a mess and a complete failure. It had duplicate code everywhere, specially for "utilities" classes. It was bloated, slow, and it was not what the customer wanted.

I'm sure that many of us can relate. We have all been on a project or a contract, where the head of the department or the team lead, decides not to refactor because it will jeopardize the deadline of the project. This is a constant tug of war that I have with some individuals. We need to take a stand and explain to these leaders that we cannot keep kicking the can. The new buzz word for this is "technical debt". There are sometimes that we can have some type of technical debt, but as I said before, there are other times that the code is just screaming for a refactor.

That's why I like the comparison of software development to gardening. You can plant new plans in your garden, but then you have to clean it up once in a while to remove all the weeds. This is the refactoring part.

Here is a list that I have made to know when we should refactor a piece of code:

• Not able to test
• Very hard to read
• Does too many things
• Too big/bloated classes or methods
• It is mutable when it shouldn't
• When the code is not DRY
• Method has over 5 parameters
• Too many nested loops or if statements

There is a great quote that I found in wikipedia:

By continuously improving the design of code, we make it easier and easier to work with. This is in sharp contrast to what typically happens: little refactoring and a great deal of attention paid to expediently adding new features. If you get into the hygienic habit of refactoring continuously, you'll find that it is easier to extend and maintain code. -- Joshua Kerievsky, Refactoring to Patterns

Getting your equals right

I am still amazed when some of the programmers either don't use the equals properly or just don't use it at all. Here is some of the examples of when to use it and the rules of thumb according to Joshua Bloch from Effective Java (my favorite Java Book).




The class is very simple, the user passes a total amount of shares and the shares that are allocated to that total. Pretty straight forward. Here is the test case.

By following these rules, here is t


This does works. But we can do better. We can compared the objects and see if they are indeed correct. This can also helps us in future projects. As you can see, the class is immutable, which means that we can easily share this object without compromising the state. We can do this by overriding the "equals" method in the Share's class. But first, we need to remember the rule that the equals method implements the equivalence relation. It is:

• Reflexive: an object must be equals to itself
• Symmetric: any two objects must agree on whether they are equal
• Transitive: if one object is equal to a second and the second object is equal to a third, then the first object must be equal to the third.
• Consistency: if two objects are equal, they must remain equal for all the time, unless one (or both) of them is modified
• Non-nullity: all objects must be unequal to the invocation of o.equals(null)

Here is the class with the equals override:

Now we can compared the actual objects:


Much better, but according to Item 9 in Effective Java, we should always override hashCode when we override equals. Failure to do so will result in a violation of the general contract of Object.hashCode which will prevent your class from functioning properly in conjunction with all hash-based collections, including HashMap, HashSet, and HashTable. The good thing is that Eclipse can be a great tool. By just clicking on Source code > Generate hashcode and equals you get this:

Start with Jenkins CI #win!

I wanted to start using Jenkins for quiet some time. I used Continuous Integrations in other projects, and they are absolutely helpful! To start Jenkins was very easy thanks to their great website. These are the steps that I used to installed Jenkins in my CentOS release 5.6 box. Jenkins automatically gets installed in the port 8080, but I have something running there and I wanted to keep record of where all my services are running so I changed that to 8009.

1 - Installation:

2 - changed the configuration by going into /etc/sysconfig/jenkins and edited the default port JENKINS_PORT="8089"

3 - restart or start jenkins by doing a "service jenkins restart" and monitor the log /var/log/jenkins/jenkins.log in case there are any errors

4 - go into your server: http://localhost:8089

This is not your grandma's iBatis...is MyBatis!

I am currently starting a new project with one of my client. The previous team created their own ORM (ugh). I've noticed that they were not closing any SQL connections which is causing performance issues. Clearly, I need to refactor the ORM class. I wave several options:

• Spring JDBC
• MyBatis (currently know as iBatis)
• Hibernate
• JPA

For big starting projects (big green projects), I usually prefer Hibernate. Even though it is heavy and resource intensive, once you set it up, it is easy to use and maintain. It is also a very mature ORM. I usually choose this tool for major re-factoring and if the project has a lot of domain object and non-complex SQL queries. For small projects, I usually prefer MyBatis (previously named iBatis). It is light and if the company has complex SQL you can just map the SQL to objects via "mappers".

This particular project, it does not have any domains and have some complex queries, so I decided to use it. However, when I went to the site, I noticed that they have moved the code and not only they changed the name (it is now mybatis instead of iBatis) but that the API completely changed. I also learned that the connection template for Spring changed. Nevertheless, I still thought that Hibernate was going to be an overkill, so I decided to give MyBatis a shot. Here is some of the the stuff I learned about the new API and its connection with Spring. The following is my software stack for this project:

• Java 1.6
• Spring 3.0
• mybatis 3.0
• mybatis-spring 1.

I will show the following on this post:

1. Configuration and Mappers
2. Connection via Spring
3. Testing using spring
4. Log4J configuration

Configuration and Mappers
Nothing much has change here with the exception of the XML headers. The aliases and the mappers do behave the same. As you noticed, I do not include any of the parameters for the connection. This will be handle by Spring.



The domain that I will be using for the mapping is the following Destination class.


The mappers have some changes to it compared to the previous iBatis. In here, I used the constructor since my destination class is immutable. As always, the select point to the mapper so that what it fetches the record(s) it know how to map the POJO.


My repository layer is very simple. As you noticed, I'm using the Repository and Qualifier to be able to autowired it.

The database configuration is basically the same with the exception of the sqlSessionFactory. As you can see I have two properties: datasource and configLocation. The configLocation is where my config.xml file is located.

There is nothing that I need to do with the DAO/Repository context since it is using the component-scan:

My DAO test is the following: