Executing a command line via Groovy to Nagios

The other day I got stuck trying to execute a Linux command via Groovy. The command needed to pass several parameters to Nagios (my monitoring system). When I was trying to pass all the parameter via String, this did not work. The way that I was able to resolve it is using a string array. Here is an example:

Find information of Stocks using Yahoo's YQL

I needed to find information about specific ticker symbols and get their stock market(NASDAQ, NYSE, etc). I found my answer with Yahoo's Query Language (YQL) - very nice API! Here is a quick Groovy application that basically goes and fetches information from the site and retrieves the ticker symbols and their correspondent exchange/stock market.


Here is the unit test cases:

Testing and Groovin'

I've come to the conclusion that writing unit tests in Java is such a pain. Specially when there are others languages that are better suited for writing test. Lately, I've been writing all my test using Groovy. I have been using it for writing unit tests and creating mock objects. I am using EasyMock for most of my programming time, but I recently found out that Groovy has its own mock object framework. Here are some example from the site. I think that they are pretty compelling.

Domain Layer:




Service Layer:







This is the class that we are going to test. As you noticed, we are using the ExchangeRateService class.


Tests:

Here we will need to mock the ExchangeRateService. We are going to use a few example that Groovy can provide us:
- You can often get away with simple maps or closures to build your custom mocks
- By using maps or expandos, we can incorporate desired behaviour of a collaborator very easily as shown here:



This is an example of using it with closure:



If we need the full power of a dynamic mocking framework, Groovy has a built-in framework which makes use of meta-programming to define the behaviour of the service. An example is shown here:

As I mentioned, just like any other mocking framework, Groovy has its own framework build-in: "MockFor" which defines the behavior of the collaborator.


Let me know how it goes and thanks for reading!

Getting your Groovy on! - Groovy 1.8

I started working on the new version of Groovy 1.8. Here is a brief description of how you can start using Groovy and the new features of Groovy 1.8. Although, this post has some basic information about Groovy, it is not a "getting started" guideline. If you are new to Groovy, I highly recommend you to go to the following sites. They cover the language in great detail and provide some good documentation:

- Bazhidar Batsov has a great post on Groovy
- Groovy has a great Getting Started Guideline

Installation and Getting Started

- Download the binary distribution and unpack it into some folder (i.e. /groovy-1.8.0)
- (Optional) Create a symbolic link to the groovy directory and I name it "/groovy".
- Set your GROOVY_HOME environment

if you do a "ls -l /groovy" you will get the following:

In my application I do the following (MacBook Pro using TextMate):

To make sure that you have the latest installation, do the following:


Most people I know work with Eclipse. Groovy has a plugin for Eclipse and a tutorial to upgrade the Groovy version inside your IDE.

Using Arrays/List:


SQL and Groovy:


Which will return the following:
-- Marcelo --
-- Antonio --
-- Jorge Luis --
-- Nestor --

In the new Groovy 1.8, there is a feature for pagination:


Where the first two states the starting point and the next two states the max record to fetch.
Output:
XTO (20040809)
DE (20040809)

Tests are perfect for Groovy. You can use the latest JUnit to run all the unit test cases as do in Java. Below I will be using some of the example using JUnit test:

Closures and Arrays/List:
Different people use Groovy in different ways, but everyone agrees that the collection API for Groovy is very powerful. Below are more examples of using List or Arrays along with closures:

As you can see, the each is a closure in Groovy and it iterates through the list. The output is the following:
3
4
5
6
7
11

Closure composition
Closure composition is to be able to combine closure with one another. In some cases you want to combine two type of functions. For example:


Trampoline
Groovy used to have error when using closures in recursive algorithms. They were able to resolve this issue with trampoline. Here is an example of using the factorial function.


Memoize Cache
Memoize is a very interesting tool for Groovy. From the Groovy's documentation:

The return values for a given set of Closure parameter values are kept in a cache, for those memoized Closures. That way, if you have an expensive computation to make that takes seconds, you can put the return value in cache, so that the next execution with the same parameter will return the same result – again, we assume results of an invocation are the same given the same set of parameter values.
There are three forms of memoize functions:
- the standard memoize() which caches all the invocations
- memoizeAtMost(max) call which caches a maximum number of invocation
- memoizeAtLeast(min) call which keeps at least a certain number of invocation results
- and memoizeBetween(min, max) which keeps a range results (between a minimum and a maximum)

Here is an example:

output is:
Fri Jun 24 13:29:15 EDT 2011
Fri Jun 24 13:29:16 EDT 2011
Fri Jun 24 13:29:16 EDT 2011
Fri Jun 24 13:29:17 EDT 2011

Maps and Default Value:
Maps now can have a default value (similar to Ruby). In the case below, if we needed to count the frequency of words in a sentence, we can start a map and store the words and the counter. By configuring the default counter, all the words will have a 0


Groove on brother!

My take on Ruby and Java

Following the advise from Debasish Ghosh, I put my list of new languages to learn and one of them was Ruby.

Why Ruby?

• Many web developers have told me great deals about the language and aboutRails.
• Puppet is a system management that I want to learn and it uses Ruby.
• Some of my pending books use Ruby for its examples - Programming Amazon Web Services
• I have a few projects in my pipeline that requires a lot of code with basic funcionality (CRUD) and a nice GUI
• I want to avoid the "gold hammer" antipattern

Questions that I had up-front:

• Is the syntax that nice? EVERY SINGLE Ruby guy swears by the syntax. What is so sexy about the syntax?
• How slow is it? I have been hearing a lot of rumors regarding the horrible GC of Ruby. Java programmers have blog that JRuby was faster than Ruby
• Multithreaded - green threads? Heard a lot from colleagues that Ruby is not well equipped for multithreaded
• How long would it take me to learn it? Try to measure the learning curve.
• Where would I use this in real life? Where are the best places/projects that I could use Ruby and most important which ones to avoid (Twitter).
• How does it compare to Java? I'm sure that there are some pros and cons regarding the syntax, but what is difference of using Groovy, JRuby, or Scala vs using Ruby?

Analysis
Syntax
Indeed, the syntax is very nice. It is very similar to Groovy. It does has a few features in the language that I haven't seen in others. Here are some examples:

Virtual Attributes

I created a BookInStock class along with a few required parameters, then I wanted to get the price in cents.

class BookInStock
attr_reader :isbn, :price
attr_writer :price

def initialize(isbn, price)
@isbn = isbn
@price = Float(price)
end

def price_in_cents
Integer(@price*100 + 0.5)
end

def price_in_cents=(cents)
@price = cents / 100.0
end
end

book = BookInStock.new("isbn1", 33.80)
puts "Price = #{book.price}"
puts "Price in cents = #{book.price_in_cents}"
book.price_in_cents = 1234
puts "Price = #{book.price}"
puts "Price in cents = #{book.price_in_cents}"

Result:
Price = 33.8 Price in cents = 3380
Price = 12.34
Price in cents = 1234

Here we've used attribute method to create a virtual instance variable. To the outside world, price_in_cents seems to ben attribute like any other. Internally, though, it has no corresponding instance variable.

Default Hash
Lets say that you need to count the amount of words in a file. The way that would do the counting, is that each line would be treated like an array of words. Create a hash, check if the hash has the word, if it doesn't then add the count of 1; otherwise, increment it.

if(counts.containsKey(word))
counts.put(word, counts.get(word) + 1);
else {
counts.put(word, 1);
}

In Ruby you can create a Hash.new(0), the parameter (0 in this case) will be used as the hash's default value - it will be the value returned if you look up a key that isn't yet there.

def count_frequency(word_list)
counts = Hash.new(0)
for word in word_list
counts[word] += 1
end
counts
end

Regular Expression
You can give name to the pattern and retrieve the value of the matched regular expression

pattern = /(\d\d):(\d\d):(\d\d)/
string  = "It is 12:34:56 precisely"
if match = pattern.match(string)
puts "Hour = #{match[:hour]}"
puts "Min  = #{match[:min]}"
puts "Sec  = #{match[:sec]}"
end

The result is the following:
Hour = 12
Min = 34
Sec = 56

Lambda
Lambda is like a call done assuming that it is like a method:
Form example:

say_hi = lambda {|a| "Hello #{a}"}
say_hi.("quintin")

Will return, "Hello quintin"

Another example, lets say that you have to calculate ten consecutive numbers but the calculation is based on a parameter passed by the user (multiplication or addition).

if operator == :multiplication && number != nil
calc = lambda{|n| n * number}
else
calc = lambda{|n| n + number}
end

puts ((1...10).collect(&calc).join(","))

Here, line 7 does the iteration of each of the 10 numbers and uses the appropriate calculation based on the parameter and passes each number to it. Then, it joins each number with a comma.

Slow?
The Ruby team has done a lot to its virtual machine, but (coming from a static typed language perspective) it is still hugs a lot resources.

Multithreaded
In the previous version of Ruby (1.8), the way that it handle the threads was through its VM. This process is called green threads. In Ruby 1.9, threading is now performed by the operating system. The advantage is that it is available for multiprocessors. However, it operates only a single thread at a time.

Learning Curve
If you are using Groovy, Python, or any similar dynamic language, the learning curve is almost null. The only problem is the lack of good support for editors. NetBeans used to be a very nice editor, but they have decided to discontinue the support for Ruby. I've been a Mac guy for quiet some time now, so I've been using TextMate and it worked great.

Where should I use it?
I try to "pigeon hole" the applications carefully before me or my team decide which type of language to use. For example, if the application needs to be highly efficient with a large amount of transactions, and performance needs to be immediate, I do NOT trust Ruby (sorry). I would turn to Java, Spring, Hibernate/iBatis, EHCache stack. However, if the application is a quick and simple CRUD pages with low usage like a series of admin pages, then Ruby would be my choice.

Comparing Ruby to Java
It is worth to keep Ruby in your toolbox in case you need to do quick scripts or sites. I did end up learning different things, like Ruby's nice components. For example, if you would like to get the top-five words in the previous word count example, you can just do the following:

sorted    = counts.sort_by {|word, count| count}
top_five  = sorted.last(5)

The sort_by and inject are two handy component. Here are some other examples:

[ 1, 2, 3, 4, 5 ].inject(:+) # => 15

( 'a'..'m').inject(:+)       # => "abcdefghijklm"

Also, being a TDD guy, I really enjoyed the Behavior-Driven Development or BDD. Based on the books and forums that I've read, this is the Ruby community's choice of tests. It encourages people to write tests in terms of your expectations of the program’s behavior in a given set of circumstances. In many ways, this is like testing according to the content of user stories, a common requirements-gathering technique in agile methodologies. Some of the frameworks are RSpec and Shoulda. With these testing frameworks, the focus is not on assertions. Instead, you write expectations.

The class that I created is a class for tennis tournament:

class TennisScorer
OPPOSITE_SIDE_OF_NET = {:server => :receiver, :receiver => :server}

def initialize
@score = { :server => 0, :receiver => 0 }
end

def score
"#{@score[:server]*15}-#{@score[:receiver]*15}"
end

def give_point_to(player)
other = OPPOSITE_SIDE_OF_NET[player]
fail "Unkown player #{player}" unless other
@score[player]  += 1
end

end

The test is the following:

require 'simplecov'
SimpleCov.start

require_relative "tennis_scorer"

describe TennisScorer, "basic scoring" do
let(:ts) { TennisScorer.new}


it "should start with a score of 0-0" do
ts.score.should == "0-0"
end

it "should be 15-0 if the server wins a point" do
ts.give_point_to(:server)
ts.score.should == "15-0"
end

it "should be 0-15 if the receiver wins a point" do
ts.give_point_to(:receiver)
ts.score.should == "0-15"
end

it "should be 15-15 after they both win a point" do
ts.give_point_to(:receiver)
ts.give_point_to(:server)
ts.score.should == "15-15"
end
end

As you can see you add context to your test and check if the solution is fine. Executing the tests echoes the expectations back and validates each test.

How slow is it?
Dave Thomas, author of the book Programing Ruby 1.9 version 3 wrote the following in his post regarding the re-factoring of the code of Twitter from Ruby to Scala:

At the kinds of volumes that Twitter handles (and with what I assume is a somewhat scary growth curve), Twitter needs to improve concurrency—it needs an environment/language with low memory overhead, incredible performance, and super-efficient threading. I don't know if Scala fits that particular bill, but I know that current Ruby implementations don't. It isn't what Ruby's intended to be. So the move away is just sound thinking. (I suspect it also took some courage.) I applaud Alex and the team for this.

Instead of defending Ruby when it's clearly not an appropriate solution, let's think about things the other way around.

The good folks at Twitter started off with Ruby because they wanted to get something running quickly, and they wanted to experiment. And Ruby gave them that. And, what's more, Ruby saw them through at least two rounds of phenomenal growth. Could they have done it in another language? Sure. But I suspect Ruby, despite the occasional headache, helped them get where they are now.

Conclusion
Finally, I would recommend learning Ruby and I would definitely keep doing more stuff with it. Although there are some characteristics similar to Groovy, Python, and others dynamic languages, it does has some nice different features. Also, the Ruby community is very vibrant/active. There are thousands of programmers building packages/APIs or gems.

The way to load the API is very similar to the "yum" command in Linux. Let say you need to do the following:

1. Connect to a GMail account
2. Check for e-mails that have attachments
3. Do some type of business logic
4. Send an e-mail with your results

I could create everything from zero, but instead I was able to find a nice little gem named "gmail". I just did "gem install gmail" and voila, I got the API! I also wanted to use MongoDB for a Ruby on Rails (RoR) project and a podcast explain to me how to do it.

Again, this is just the beginning but it was a really nice experience and remind me back of why I got into programming. The challenge and the unknown is what drive must of us on finding better solutions.

Run all your Groovy test cases using Ant

I needed to run all my Groovy unit test cases using Junit ant task, but for whatever reason I was not able to do it. I was able to do it using the following:

Marshalling XML- Castor vs. Groovy MarkupBuilder

I worked on a small project where I needed to use some type of XML marshalling. I end up using Spring's (version 3) integration with Castor and JMS (ActiveMQ). I find it not that much flexible and eventually ended up using Groovy's XML libraries (really nice).

We had problems using the "conversion" of JMS (from object to JMS message), we find it easier to use an XML to transfer the messages in the JMS queue. The object that I wanted to convert to XML was a value object (immutable no setters) and using static factory. The object is the following:

public final class Redirection {

private RegisteredDate registeredDate;
private Authentication authentication;
private OperatorAndService operatorService;
private TextMessage textMessage;

private Redirection() {}


private Redirection(Authentication authentication,
OperatorAndService operatorService, TextMessage textMessage, Date date) {
this.authentication = authentication;
this.operatorService = operatorService;
this.textMessage = textMessage;
registeredDate = RegisteredDate.get( date!=null?date:new Date());
}

...

public static Redirection get(Authentication authentication,
OperatorAndService operatorService, TextMessage textMessage, Date date) {
return new Redirection(authentication, operatorService, textMessage, date);
}

...

public Authentication getAuthentication() {
return authentication;
}

public OperatorAndService getOperatorAndService() {
return operatorService;
}

public TextMessage getTextMessage() {
return textMessage;
}

public RegisteredDate getRegisteredDate() {
return registeredDate;
}

@Override
public String toString() {
return "Authentication: " + authentication + ", operator service: "
+ operatorService + ", text message: " + textMessage +
", registed date: " + registeredDate;
}
}

The implementation of the Castor is pretty simple using Spring 3:

I used the mapping, since I needed to pass three objects to the static factory to create my Redirection object (Authentication, OperatorAndService, and Textmessage). The mapping file was the following:

 Mapping for the redirection object

 
 
 
 

 
      
           
      
      
           
      
      
           
      
      
           
      
 
    

The "get-method" is exactly what it means, is the method that you will use to fetch the object. The set-method is in case you have any setters (perfect for DTOs). However, my class is immutable so there are no setters. Instead, there is one static method that contains the three parameters that I needed to pass. There is a way to create an object without a constructor "verify-construcable=false". I wanted to see if there is a way to pass all the four object to my static factory, however, this did not work. The creation of the XML worked but not the creation of the object from the XML. I got the following error:

Castor unmarshalling exception; nested exception is org.exolab.castor.xml.MarshalException: unable to find matching public constructor for class

I try to look for a solution, but I had to tweak a lot of things to create a Redirection object. I then decided to look at Groovy's MarkupBuilder.

class XmlService {
private XmlService() {}

public static String getXml(Redirection redirect) {
if(redirect == null) {
throw new IllegalArgumentException("Redirection is invalid or null")
}

Authentication credentials = redirect.getAuthentication()
OperatorAndService operAndServ = redirect.getOperatorAndService()
TextMessage sms = redirect.getTextMessage()

def xml = new groovy.xml.StreamingMarkupBuilder()
xml.encoding = "UTF-8"
def redirection = {
mkp.xmlDeclaration()
redirection() {
authentication() {
username(credentials.getUsername())
password(credentials.getPassword())
}
operatorAndservice() {
serviceId(operAndServ.getServiceId())
operatorId(operAndServ.getOperatorId())
}
textmessage(){
phone(sms.getPhone())
shortcode(sms.getShortcode())
message(sms.getMessage())
}
registerdate(date:redirect.getRegisteredDate())
}
}

return xml.bind(redirection);
}


public static Redirection toRedirection(String xml) {
if(Validation.isEmpty(xml)) {
throw new IllegalArgumentException("The application needs to have an xml")
}

def redirectionXml = new XmlParser().parseText(xml)
//Authentication
String username = redirectionXml.authentication.username.text()
String password = redirectionXml.authentication.password.text()
def authentication = Authentication.get(username, password)

//Operator and service
String serviceId = redirectionXml.operatorAndservice.serviceId.text()
String operatorId = redirectionXml.operatorAndservice.operatorId.text()
def operatorAndService = OperatorAndService.get(operatorId, serviceId)

//Text message
String phone = redirectionXml.textmessage.phone.text()
String shortcode = redirectionXml.textmessage.shortcode.text()
String message = redirectionXml.textmessage.message.text()
def textMessage = TextMessage.get(phone, shortcode, message)

String registerDate = redirectionXml.registerdate[0].'@date'
def registeredDate = RegisteredDate.get(registerDate)

return Redirection.get(authentication,
operatorAndService, textMessage,
registeredDate.getRegisteredDate());
}
}

Groovy is a better candidate for this application. Its simplicity and flexibility beats Castor. The only thing that concerned me is the performance. I did noticed that the test cases using Castor were much faster. In case that performance is an issue, consider using Castor or other type of XML marshaling. I heard from one of my coworkers that xstream is very fast.

Groovy on Grails

I've been playing with Groovy and Grails for the past month. We had a demand of a few sites that needed to be constructed. The sites were small and needed to have some basic CRUD method and a small business rules. Then, we had to tie it with our SMS Gateway that we have developed. We evaluated a few languages, but at the end we chose Groovy for the following reasons:

1. Dynamic language
2. It sits in the JVM
3. Easy to develop sites

We have been playing with the following versions:

• Grails 1.3.3
• Groovy 1.6.0_20

I have the most respect for the team who build this programming language. The language is very easy to read and the learning curve is very small. The integration that they did with Java, Spring, and Hibernate is very nice. The ORM tool that they created, GORM is very nice and very easy and quiet frankly it's my favorite.

class User {

String userId
String password
String homepage
Date dateCreated
Profile profile
...

}

User has a one to one relation with profile.

class Profile {
static belongsTo = User

byte[] photo
String fullName
String bio
String homepage
String email
String timezone
String country
String jabberAddress

   static constraints = {
 fullName(nullable: true)
 bio(nullable: true, maxSize: 1000)
 homepage(url: true, nullable: true)
 email(email: true, nullable: true)
 photo(nullable: true)
 country(nullable: true)
 timezone(nullable: true)
 jabberAddress(email: true, nullable: true)

   }

String toString() {
 "Profile for ${fullName} (${id})"
}
}

The belongsTo is assigned the owning class, which means the relationship is unidirectional. You can get to a Profile via a User, but there’s no link back from a Profile to a User

class Post {
String content
Date dateCreated

   static constraints = {
 content(blank: false)
   }
static belongsTo = [user: User]

static mapping = {
 profile lazy: false
 sort dateCreated: "desc"
}

static hasMany = [tags: Tag]
}

You can sort automatically within the mapping closure. In this example, all queries to the Post object will return in a descending order.

We use the map style of belongsTo, where we create a bidirectional link between User and Post classes. This creates a new field on Post called user that is the bidirectional mapping back to the owning User

Another useful technique is the validation of the fields. In the constraint closure you can specify the fields that need to be validated and the validation type. For example, in this case, content cannot be blank (null or empty).

The first deployment that I made was using "grails war". The con of this approach is that I look the dynamic compilation. So, if any change to the code, you would have to create the war once again, and deploy it. You gain performance but you loose the dynamism of Groovy. If performance is not a big issue, you can use "grails run-war". According to the documentation:

This is very similar to the previous option, but Tomcat runs against the packaged WAR file rather than the development sources. Hot-reloading is disabled, so you get good performance without the hassle of having to deploy the WAR file elsewherthe application is compiled with Java native code. Therefore, every change that I have to make I have to do it in Grails and deploy it once again.

I welcome your feedback in case I'm wrong.