But first, let’s talk about defensive coding and how we do it. For example:

/**
 * frobs the bar and baz.
 * @param bar the bar to frob; may not be null
 * @param baz the baz to from; must be positive
 */
public String frob(String bar, int baz) {
  if (bar.length() > baz) {
    return bar.substring(baz);
  }
  else {
    return bar;
  }
}

If the contract is violated by the caller, we will get a NullPointerException or some other StringIndexOutOfBoundsException. Not too helpful. Instead, we use an internal validator helper class to provide useful messages. This is crucial for properly understanding bugs that occur in production; there’s nothing worse than coming into work to find a NullPointerException in your inbox and no clue what went wrong.

/**
 * frobs the bar and baz.
 * @param bar the bar to frob; may not be null
 * @param baz the baz to from; must be positive
 */
public String frob(String bar, int baz) {
  Validate.notNull(bar,"bar must not be null to frob");
  Validate.isTrue(baz > 0,"baz must be positive");
  if (bar.length() > baz) {
    return bar.substring(baz);
  }
  else {
    return bar;
  }
}

Validate.* methods essentially throw IllegalArgumentException if what they are checking for fails.
We use this pattern extensively, even in constructors of simple data-structure style objects. This (plus keeping this things immutable) allows users of these objects to safely rely upon the get methods behaving properly. I had need of a class to hold both a Person (representing a website user) and a Customer (representing a utility-company customer). This class (called, naturally, PersonAndCustomer), takes both objects in its constructor and requires that they are both non-null. As a further sanity check, I wanted to make sure that the Person’s customerId matched the id of the Customer that was being passed in (i.e. that they represented the same actual human in the real world):

public PersonAndCustomer(Person p, Customer c) {
  Validate.notNull(p,"Person may not be null");
  Validate.notNull(c,"Customer may not be null");
  Validate.isTrue(p.getCustomerId() == c.getId(),
    "Person %d's customer id %d didn't match passed-in customer's id %d",
    p.getId(),
    p.getCustomerId(),
    c.getId());
  ...
}

Looks pretty inocuous, right? Well, because these objects are Hibernate-managed, all of our ids are Long and not long. So, my isTrue validation is really checking that the person’s customerId object is the same object as the customer’s id object. What’s worse, in several months of development and production deployment, these two objects happened to be the same.

Until this week; we were testing our application for a new client and I got the error posted above (“Person 1001’s customer id 109032 didn’t match the passed-in customer’s id of 109032″). It seems that much like how the JVM will re-use string objects, it also will sometimes re-use boxed objects as well. But not always. The fix for this is obvious, but I wanted to make sure I could actually recreate this situation. So, I created a test that gave both the Person and Customer the same value for the customer id, but as different objects. and verified that the class could still be constructed. Sure enough, the test failed. The fix:

public PersonAndCustomer(Person p, Customer c) {
  Validate.notNull(p,"Person may not be null");
  Validate.notNull(c,"Customer may not be null");
  Validate.isTrue(p.getCustomerId().equals(c.getId()),
    "Person %d's customer id %d didn't match passed-in customer's id %d",
    p.getId(),
    p.getCustomerId(),
    c.getId());
  ...
}

What’s interesting is that if one end of the == is a primitive, the JVM will unbox the other one and the test succeeds:

Long l1 = new Long(45L);
Long l2 = new Long(45L);

System.out.println(l1 == l2);      // false
System.out.println(l1.equals(l2)); // true
System.out.println(l1 == 45L);     // true!

The moral of the story is to always know what you are comparing. Might even be best to always use .equals() unless the compiler complains.

My first thought was: there’s lots of blocking reading and writing from the database, so adding some threads should speed things up. While one thread was analyzing Bob’s energy data, another could be fetching Mary’s, while another could be updating Joe’s meta-data with the results. Or so I thought.

The more threads I added, the slower the entire thing became. It turned out that the fastest implementation was a single-threaded one. But why? It all has to do with the diminishing returns one gets from scaling out.

If you think of a task as having a serial component, which cannot be broken up concurrently, and then multiple tasks which can be done concurrently, we can analyze the returns we get by increasing the number of available processors (threads, in my case). This is Amdahl’s Law and is exemplified by the following equation (where “x” is the number of processors or threads, and “s” is the percentage of your overall task that must be serialized; “y” is the increase in speed you will see from scaling).

When you graph this, it’s pretty obvious that there are diminishing returns to adding more threads/processors (the graph below assumes that 90% of the overall job can be done concurrently). As we add threads, we get less and less of a gain in speed.
But there’s still a gain to get, so what happened to me?

Amdahl’s law is actually pretty optimistic. It doesn’t account for the overhead required to synchronize the shared data. If we account for this, with a new value “k” ( the percentage penalty for maintaining consistency), we see that increasing our processors/threads actually starts to hurt us (this equation is in red)!

In my case, almost all of this synchronization is happening within the database; it turns out that almost all the time my process needs is accessing data from the database. So, I dialed it down to only one thread, and we should be done early next week.

Nacho Cheese Doritos.

Honorable Mentions: Fritos and Cool Ranch Doritos.

The easiest way to do that is to look at the bugs our QA staff find in the product updates. Although we have a lot of automated tests, we still need some eyes on the applications to check for things like text overruns of our printed reports, or JavaScript weirdness in some less-than-well-behaved) browsers.

After a few iterations, Anh (our beloved program manager) ran some queries on our bug-tracking database and, using some Excel magic, showed us numbers like “number of defects in the web application” and “average number of days a critical bug sat unassigned”. This is good stuff; seeing hard data about what we’re doing lets us focus on inefficiency. Since we release software every month (and launch new clients almost as frequently), this is a huge help in making each iteration better than the previous.

While Anh’s hand-jammed report was awesome, manually connecting SQLite and Excel is no fun for anyone. I whipped up a basic Ruby application using Sinatra to automate most of what we’d like to see. We can now see how long critical defects site open and unfixed, as well as how many overall defects we’re fixing and even the root source of our defects (e.g. did we get the wrong requirements, or just mess up implementing them?)

We still need to show trends, however. “Classic” thinking in software process improvement says to do that by dividing everything by lines of code; if we had 10 defects per 1,000 lines of code this month, but only 8 the following, we’re doing better. This is not a great measure for us, since we are changing and enhancing our product (as opposed to writing fresh code each time), so we decided to normalize our measurements by the story points we assigned to the iteration. This way, we can see charts like the one below (which says we are doing better each time):

This information is dead simple to collect; the burden on developers and testers when writing defects is almost nothing. With only a priority, severity, and “source of defect”, we can collect a lot of advanced information about our development process — and make it better iteration by iteration.