Scalability and Performance - learning from the best (Amazon and eBay)

I always go back to these presentations at least once a year to review performance and scalability.

• Availability & Consistency
• Randy Shoup on eBay's Architectural Principles
• eBay Architecture
• Scalability Best Practices: Lessons from eBay

Amazon and eBay are architectures that makes any developer salivate or wonder, "how they do it?". The Werner Vogels and Randy Shoup are (at least to this humble developer) awesome architects which we can learn MUCH about what they do best; architect high available, high performance, and highly available architectures. Werner Vogels discusses the CAP theorem very well, and how it's a KILLER to current applications. Shoup discusses the forces that take place when you develop these type of architectures.

GMail and log4j e-mail appender - error STARTTLS

I couldn't get log4j working along with Google. I kept on getting the following error:

com.sun.mail.smtp.SMTPSendFailedException: 530 5.7.0 Must issue a STARTTLS

Later I found out that the best way to get it working was creating my own SMTP appender. What I wanted, was to have a flexible application that I could configure once with a gmail username (marcelo@gmail) and then have the users/clients configure the rest of the log4j appender using the log4j property file.

Below is how I created the application. First you will need your parameters for your gmail account including the starttls. I place all these in a Enum class.

public enum EmailEnum {
  HOST("smtp.googlemail.com"), PORT("587"),
  USERNAME("marcelo@gmail.com"),PASSWORD("secret"),
  AUTHORIZED("mail.smtp.auth"), STARTTTS("mail.smtp.starttls.enable"),
  TRUE("true");
  private String value;
  private EmailEnum(String value) {
      this.value = value;
  }

  public String getValue() {
      return value;
  }
}

Then, I created the class that actually creates the mail appender. The main part is the appender method. As you can see, I am using the SMTPAppender class and I override the appender method. As you can tell, I'm using the Spring SimpleMail class to send the message.

package com.upmobile.midccore.commons.logger;

import java.util.Properties;

import org.apache.log4j.net.SMTPAppender;
import org.apache.log4j.spi.LoggingEvent;
import org.springframework.mail.MailException;
import org.springframework.mail.SimpleMailMessage;
import org.springframework.mail.javamail.JavaMailSenderImpl;


/**
* Application builds the SMPT Appender for the Google Mail (gmail).
* @author marceloolivas
*
*/
public class GMailAppender extends SMTPAppender {
   private JavaMailSenderImpl javaMail;
   private static final String NL = System.getProperty("line.separator");
  
   public GMailAppender() {
       javaMail = new JavaMailSenderImpl();
       javaMail.setHost(EmailEnum.HOST.getValue());
       javaMail.setPort(Integer.parseInt(EmailEnum.PORT.getValue()));
       javaMail.setUsername(EmailEnum.USERNAME.getValue());
       javaMail.setPassword(EmailEnum.PASSWORD.getValue());
       Properties props = new Properties();
       props.setProperty(EmailEnum.AUTHORIZED.getValue(), EmailEnum.TRUE.getValue());
       props.setProperty(EmailEnum.STARTTTS.getValue(), EmailEnum.TRUE.getValue());
       javaMail.setJavaMailProperties(props);
      
   }
  
   @Override
   public void append(LoggingEvent event) {
       super.append(event);
       // Create a thread safe "copy" of the template message and customize it
        SimpleMailMessage msg = new SimpleMailMessage();
       StringBuilder builder = new StringBuilder();
       builder.append(getLayout().format(event));
       builder.append(event.getMessage().toString());
       if (event.getThrowableInformation() != null) {
           builder.append(NL);
           String[] stackTrace = event.getThrowableInformation().getThrowableStrRep();
           for(int i = 0; i < stackTrace.length; i++) {
               builder.append(stackTrace[i] + NL);
           }
           msg.setText(builder.toString());
       }
       String[] senders = getTo().trim().replace(" ", "").split(",");
       msg.setTo(senders);
       msg.setSubject(this.getSubject());
       try{
           javaMail.send(msg);
       } catch (MailException ex){
       }
      
   }
}

The actual log4j properties file is the following:

# Global logging configuration
log4j.rootLogger=ERROR, stdout, EMAIL
# SqlMap logging configuration...
log4j.logger.com.ibatis=DEBUG
log4j.logger.com.ibatis.common.jdbc.SimpleDataSource=DEBUG
log4j.logger.com.ibatis.sqlmap.engine.cache.CacheModel=DEBUG
log4j.logger.com.ibatis.sqlmap=DEBUG
log4j.logger.com.ibatis.sqlmap.engine.builder.xml.SqlMapParser=DEBUG
log4j.logger.com.ibatis.common.util.StopWatch=DEBUG
log4j.logger.java.sql.Connection=DEBUG
log4j.logger.java.sql.Statement=DEBUG
log4j.logger.java.sql.PreparedStatement=DEBUG
log4j.logger.java.sql.ResultSet=DEBUG

log4j.logger.org.springframework=INFO
log4j.logger.com.upmobile=DEBUG
# Console output...
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%-4p [%d{ISO8601}][%t] (%c.%M()line %-4L ) %m%n

# e-mail appender
log4j.appender.EMAIL=com.upmobile.midccore.commons.logger.GMailAppender
log4j.appender.EMAIL.Threshold=ERROR
log4j.appender.EMAIL.Subject=Error with the application
log4j.appender.EMAIL.to=abc@acme.com,xyz@acme.commons
log4j.appender.EMAIL.SMTPDebug=true
log4j.appender.EMAIL.layout=org.apache.log4j.PatternLayout
log4j.appender.EMAIL.layout.ConversionPattern=[%d] [%t] %-5p %c %x - %m%n
log4j.appender.EMAIL.BufferSize=1

Motivating programmers

Dan Pink had an extraordinary presentation in TED regarding the disconnect on traditional reward system. We have learned over and over about the power of incentives and how they should improved motivation. However, science has proven that for complex problem this does not work. Instead, it could dull thinking and it slows creativity. It was impressive that for the past forty years what science has proven the business has not noticed.

Contingent motivator (cause and effect) it's a good approach for 20th century tasks for simple set of rules (narrow our focus), but for most jobs/problems (specially programmers/software engineers) extrinsic motivators are better because we need to see the entire context (work with our left and right brain). He explained it extremely well using the "candle problem".

I can watch this presentation over and over, and always find something useful. Highly recommend it.

Eliminating obsolete objects reference with profiling tools

I'm getting ready for my Miami Java User Group (MJUG) talk on December with buddy Antonio Llanos. We thought about doing a "performance and scalability" talk. I wanted to focused on the tools (specially open-source) that a programmer should have to help them analyze their applications. I also wanted for the attendants to come out of the talk knowing exactly what they should do the next day.

I also wanted to come up with snippet of code that would not be so easy to detect to show the importance of these tools. It was then when I stumbled with a piece of code from Effective Java 2nd Edition, "Item 6: Eliminate obsolete objects references". The code is a simple stack class (LIFO algorithm), which has a memory leak.

package com.midc.spikes.effectivejava.obsoleteobject;

//Can you spot the "memory leak"?

import java.util.*;

public class Stack {
private Object[] elements;
private int size = 0;
private static final int DEFAULT_INITIAL_CAPACITY = 16;

public Stack() {
   elements = new Object[DEFAULT_INITIAL_CAPACITY];
}

public void push(Object e) {
   ensureCapacity();
   elements[size++] = e;
}


public Object pop() {
   if (size == 0)
       throw new EmptyStackException();
   return elements[--size];
}

/**
*
* Ensure space for at least one more element, roughly
* doubling the capacity each time the array needs to grow.
*/

private void ensureCapacity() {
   if (elements.length == size)
       elements = Arrays.copyOf(elements, 2 * size + 1);
}

public int getSize() {
   return size;
}
}

I couldn't find the memory leak. According to the book,

If a stack grows and then shrinks, the objects that were popped off the stack will not be garbage collected, even if the program using the stack has no more references to them.

To fix the problem, we just need to null the reference of the object that is popped.

    public Object pop() {
   if (size == 0)
       throw new EmptyStackException();
   Object result = elements[--size];
   elements[size] = null;
   return result;
}

The book also explains, that these bugs are hard to detect, and that most the best way to find them is through "careful code inspection or with the aid of a debugging tool known as a heap profiler". As always, it is very desirable to find these type of problems ahead of time to prevent them from happening. In other words, exactly what I needed for my talk.

Ruminations of a Programmer: 5 Reasons why you should learn a new language NOW!

Ruminations of a Programmer: 5 Reasons why you should learn a new language NOW!

Application Performance Management

I have been working on a topic for the Miami Java User Group. Lately, I have been doing a great deal of performance and scalability for my projects. Thanks to my friend Edson Cimionatto, he let me borrow the Apress' Pro Java EE 5 by Steven Haines. It is a nice book and I have been reading part of the chapters, so today I reviewed some of my notes. Looking back there are four things that really cought my attention:

1. Have SLAs
2. Performance applies the rule of "better to do it from the start" or "earlier is cheaper"
3. Have a set of tools to diagnose the application (memory, code profiling, and code coverage)
4. There is a "culture" aspect on tackling performance. Not only you need to do tests, and have a decent architecture, but you have to be willing to be relentless on keep the performance metrics (SLAs)