But first, let’s talk about defensive coding and how we do it. For example:

/**
 * frobs the bar and baz.
 * @param bar the bar to frob; may not be null
 * @param baz the baz to from; must be positive
 */
public String frob(String bar, int baz) {
  if (bar.length() > baz) {
    return bar.substring(baz);
  }
  else {
    return bar;
  }
}

If the contract is violated by the caller, we will get a NullPointerException or some other StringIndexOutOfBoundsException. Not too helpful. Instead, we use an internal validator helper class to provide useful messages. This is crucial for properly understanding bugs that occur in production; there’s nothing worse than coming into work to find a NullPointerException in your inbox and no clue what went wrong.

/**
 * frobs the bar and baz.
 * @param bar the bar to frob; may not be null
 * @param baz the baz to from; must be positive
 */
public String frob(String bar, int baz) {
  Validate.notNull(bar,"bar must not be null to frob");
  Validate.isTrue(baz > 0,"baz must be positive");
  if (bar.length() > baz) {
    return bar.substring(baz);
  }
  else {
    return bar;
  }
}

Validate.* methods essentially throw IllegalArgumentException if what they are checking for fails.
We use this pattern extensively, even in constructors of simple data-structure style objects. This (plus keeping this things immutable) allows users of these objects to safely rely upon the get methods behaving properly. I had need of a class to hold both a Person (representing a website user) and a Customer (representing a utility-company customer). This class (called, naturally, PersonAndCustomer), takes both objects in its constructor and requires that they are both non-null. As a further sanity check, I wanted to make sure that the Person’s customerId matched the id of the Customer that was being passed in (i.e. that they represented the same actual human in the real world):

public PersonAndCustomer(Person p, Customer c) {
  Validate.notNull(p,"Person may not be null");
  Validate.notNull(c,"Customer may not be null");
  Validate.isTrue(p.getCustomerId() == c.getId(),
    "Person %d's customer id %d didn't match passed-in customer's id %d",
    p.getId(),
    p.getCustomerId(),
    c.getId());
  ...
}

Looks pretty inocuous, right? Well, because these objects are Hibernate-managed, all of our ids are Long and not long. So, my isTrue validation is really checking that the person’s customerId object is the same object as the customer’s id object. What’s worse, in several months of development and production deployment, these two objects happened to be the same.

Until this week; we were testing our application for a new client and I got the error posted above (“Person 1001’s customer id 109032 didn’t match the passed-in customer’s id of 109032″). It seems that much like how the JVM will re-use string objects, it also will sometimes re-use boxed objects as well. But not always. The fix for this is obvious, but I wanted to make sure I could actually recreate this situation. So, I created a test that gave both the Person and Customer the same value for the customer id, but as different objects. and verified that the class could still be constructed. Sure enough, the test failed. The fix:

public PersonAndCustomer(Person p, Customer c) {
  Validate.notNull(p,"Person may not be null");
  Validate.notNull(c,"Customer may not be null");
  Validate.isTrue(p.getCustomerId().equals(c.getId()),
    "Person %d's customer id %d didn't match passed-in customer's id %d",
    p.getId(),
    p.getCustomerId(),
    c.getId());
  ...
}

What’s interesting is that if one end of the == is a primitive, the JVM will unbox the other one and the test succeeds:

Long l1 = new Long(45L);
Long l2 = new Long(45L);

System.out.println(l1 == l2);      // false
System.out.println(l1.equals(l2)); // true
System.out.println(l1 == 45L);     // true!

The moral of the story is to always know what you are comparing. Might even be best to always use .equals() unless the compiler complains.

We here at OPOWER use Alfresco Community 3.2 as a glorified “XML editing tool” that was easy to setup, configure and hand over to the business to allow them to contribute XML-based content that our applications can consume. As a F/LOSS solution, it’s fit our needs pretty well and is generally a low-maintenance piece of our infrastructure.

I attribute my “low maintenance” characterization of an Alfresco installation mostly to the patience of my small user base, who are very accommodating to the quirks and limitations of the installed (and neglected) instance.

The 3.3 lunch and learn was intro’d by Chris from SiteWorx who surveyed the CMS landscape and equated the emerging CMIS spec as “content’s ODBC” which I thought to be a quite fitting analogy.

Richard Im from Alfresco then spoke for about 45 minutes about the new features Alfresco 3.3 contained:

WCM Improvements

• new web-based editor, reminiscent of Interwoven’s in-context editor from the 6.5/7.0 era
• new bi-directional deployment capabilities (including addition of “delivery server” infrastructure possibilities like several other CMS offerings)
• “Straight through” publishing without workflows
• google-like search on metadata attributes

Share

• Evolving share into more of a bit-for-bit replacement of the “traditional” UI (which Richard referred to as “DM”… “document manager?”)
• Permission control a la DM
• Rules and Actions just like in DM
• Data Lists (like Share Point’s project lists)

I asked the question: “what’s the medium-term road map for DM, if Alfresco seems intent on mirroring all(?) functionality in the traditional UI via the /share context?”  Richard wouldn’t definitively say that the traditional app is going to be deprecated any time soon, but it certainly appears that way…

“Integration” improvements

I think this suite of improvements is geared towards the enterprise crowd:

• Full CMIS 1.0 spec implemented as of 3.3 (the only open source implementation known to do so)
• ACLs
• Change Logs
• Additional SOA/REST services and support
• Ability to save off google docs in alfresco for management (!)

My big take-aways from the lunch and Q&A session was that Alfresco’s 3.3 feature list and 3.4 direction seem to put them on more of a competitive footing with monster ECM vendors like Interwoven.  It’s maybe an odd direction for an open source project to steer because while Alfresco certainly employs their own development team, I’m not sure how much community interest there is in contributing to some of the more mundane enterprise ECM concerns that need to be built or integrated.  On the other hand, Alfresco, Inc. does survive with enterprise support contracts that smaller shops like mine are unlikely to ever pay top-dollar for, so maybe it’s a long term survival strategy.

Long term, I’m interested in exploring architectures available to us with a SOA-enabled CMIS layer and eventually ditch the custom-DAO layer we’re running right now against Alfresco-exported XML flat files.  Even if that never happens, it’s exciting to see the kinds of complexity that open source projects are able to attack.

My first thought was: there’s lots of blocking reading and writing from the database, so adding some threads should speed things up. While one thread was analyzing Bob’s energy data, another could be fetching Mary’s, while another could be updating Joe’s meta-data with the results. Or so I thought.

The more threads I added, the slower the entire thing became. It turned out that the fastest implementation was a single-threaded one. But why? It all has to do with the diminishing returns one gets from scaling out.

If you think of a task as having a serial component, which cannot be broken up concurrently, and then multiple tasks which can be done concurrently, we can analyze the returns we get by increasing the number of available processors (threads, in my case). This is Amdahl’s Law and is exemplified by the following equation (where “x” is the number of processors or threads, and “s” is the percentage of your overall task that must be serialized; “y” is the increase in speed you will see from scaling).

When you graph this, it’s pretty obvious that there are diminishing returns to adding more threads/processors (the graph below assumes that 90% of the overall job can be done concurrently). As we add threads, we get less and less of a gain in speed.
But there’s still a gain to get, so what happened to me?

Amdahl’s law is actually pretty optimistic. It doesn’t account for the overhead required to synchronize the shared data. If we account for this, with a new value “k” ( the percentage penalty for maintaining consistency), we see that increasing our processors/threads actually starts to hurt us (this equation is in red)!

In my case, almost all of this synchronization is happening within the database; it turns out that almost all the time my process needs is accessing data from the database. So, I dialed it down to only one thread, and we should be done early next week.

I recently converted the attribute of our Person POJO that stored a user’s email address from a java.lang.String to a javax.mail.InternetAddress.  Call it not enough testing or lack of imagination, but I introduced a Null Pointer Exception in a back-office web flow where a new user is created for our customer service application.  When a new CSR account is created and the email address is left blank, an NPE gets thrown up.  Embarrassing, but easy to fix, right?

Just looking at the stack trace narrows it down immediately:

2009-09-09 16:56:53,636 WARN  [btpool0-1] [REPORT] [WARN] Handler execution resulted in exception
java.lang.NullPointerException
	at javax.mail.internet.InternetAddress.parse(InternetAddress.java:609)
	at javax.mail.internet.InternetAddress.parse(InternetAddress.java:569)
	at javax.mail.internet.InternetAddress.(InternetAddress.java:105)
	at poscore.db.InternetAddressUserType.deepCopy(InternetAddressUserType.java:93)
	at org.hibernate.type.CustomType.deepCopy(CustomType.java:179)
	at org.hibernate.type.TypeFactory.deepCopy(TypeFactory.java:374)

Ahh…yeah…that custom hibernate type I wrote to translate VARCHARs to InternetAddresses and vice versa. Ok, what does that deep copy method look like?

    public Object deepCopy(Object value) throws HibernateException {
        if(value == null) {
            return null;
        }
        InternetAddress original = (InternetAddress)value;
        InternetAddress copy = null;
        try {
            copy = new InternetAddress(original.getAddress());
        }
        catch (AddressException ex) {
            throw new HibernateException("Unable to deep copy email address '" + original.getAddress() + "'");
        }
        return copy;
    }

It turns out, the error is right here:

            copy = new InternetAddress(original.getAddress());

You can’t pass a null to the InternetAddress constructor. It was late in the day and I must not have internalized that there’s no way value or original could be null, because my first instinct was that the NPE was actually the result of calling “.getAddress()” on a null original object….not that the constructor can’t take a null.

While I was on that assumption (that it was the original object that was null), I fired up my debugger and got confused by what appeared to be null-checks failing to check nulls. Check out this screen shot of the NPE about to be thrown after what appears to be 2 null checks failing to prevent the NPE:

The arrows marked (A) and (B) shows what appear to be null objects passing null checks (in the top pane) even though they’re being reported as null (in the bottom pane). The green line (C) shows the code falling through to that line just before it throws up the NPE.

Of course, value and original aren’t null at all…it’s just that their .toString() methods report them to be null. Here’s the InternetAddress.toString method:

  276       public String toString() {
  277   	if (encodedPersonal == null && personal != null)
  278   	    try {
  279   		encodedPersonal = MimeUtility.encodeWord(personal);
  280   	    } catch (UnsupportedEncodingException ex) { }
  281
  282   	if (encodedPersonal != null)
  283   	    return quotePhrase(encodedPersonal) + " <" + address + ">";
  284   	else if (isGroup() || isSimple())
  285   	    return address;
  286   	else
  287   	    return "<" + address + ">";
  288       }

In my situation, I fell in to the “else if(isGroup() || isSimple())” case, which returns address, which is null, so the .toString() for the whole InternetAddress object is “null.”

I think the toString() method could benefit from another null-check right at line 284, which would mean an InternetAddress object with a null internal address String would display as “<null>”…that’s a more immediate clue that the object itself isn’t null.

If you throw an exception about some connection being refused, you’d better damn well put what connection you were attempting to make in the exception message.

What inspired this law?  Thanks for asking.

Check out this snippet in the middle of an 80 line stack dump:

2009-08-25 17:23:42.373::WARN:  Nested in org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'coreAutopatcher' defined in class path resource [config/migrationContext.xml]: Invocation of init method failed; nested exception is com.tacitknowledge.util.migration.MigrationException: Error applying patches:
com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1319)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)

“A ResourcePool could not acquire a resource from its primary factory or source”. Awesome. Sooo helpful. /wristsemoragegrowl

Also, a WARN? Seriously? You’re a connection pool. Do you think not connecting to your underlying datasource warrants maybe something as stern as an ERROR? Or, heaven forbid, a FATAL?

I’m sure I could have turned up logging somewhere, but I noticed Spring’s DelegatingDataSource bean standing out from among the stack lines and set a break point on its called method to figure out exactly which of my many connections was unable to be resolved:

There’s the culprit! localhost:5455. You know, whoever the idiot is who runs that localhost server needs to get his act together. I should call the guys from YouTube and have them do a Maury Povich-style video expose on why he can’t keep any of his connections stable.

svn co http://domain.com/svnroot/project/trunk project

That way, when I go to the project directory, I do not have to then go into the trunk directory to get to my files. Saving an extra directory level is not the best benefit though. When I set up my IDE (yes, I use an IDE), I put the IDE files in that one directory and do not have to recreate them for each branch or tag that I might check out. I essentially treat the “trunk/branches/tags” part of the directory as pure metadata, which makes perfect sense to me.

When I switched over to this system, there was one big drawback. I could not easily tell where I am just by looking at my prompt. I had to run svn info | grep URL to see if I was on a branch. So instead of letting my prompt handicap me, I decided to get smart about my prompt. Using bash, my prompt used to look like this:

jeff:/opt/pose/main/report/src/main/java

but now it looks like this (supercharged with Subversion metadata):

jeff:/opt/pose/main/report/src/main/java [SVN: /main/report/trunk]

I’m no bash expert, so I dug around on the web and learned a bit about how PROMPT_COMMAND works. Here is what I have in my .bash_profile now:


function spwd ()
{
stat .svn > /dev/null 2>&1;
if [ "$?" == "0" ]; then
SURL=`svn info | grep URL | perl -pe 's/URL: (.*)/\1/'`;
if [ `echo ${SURL} | grep -E "branches|tags"` ]; then
SVER=`echo ${SURL} | perl -pe 's{.*/(branches|tags)/(.*)}{\1/\2}' | cut -d/ -f1-2`;
SPTH=`echo ${SURL} | perl -pe 's{.*svnroot/(.*)/(branches|tags)/.*}{/\1}'`;
SPWD="${SPTH}/${SVER}";
else
SPWD=`echo ${SURL} | perl -pe 's{.*svnroot/(.*)/trunk(.*)}{/\1/trunk}'`;
fi;
export PS1="\u:\w [SVN: $SPWD]\n$ ";
else
export PS1="\u:\w $ ";
fi
}
export PROMPT_COMMAND=spwd


It may not be the most efficient way to do it, but it sure works well and lets me know in an instant what trunk, branch or tag I happen to be working on at that very moment.

The easiest way to do that is to look at the bugs our QA staff find in the product updates. Although we have a lot of automated tests, we still need some eyes on the applications to check for things like text overruns of our printed reports, or JavaScript weirdness in some less-than-well-behaved) browsers.

After a few iterations, Anh (our beloved program manager) ran some queries on our bug-tracking database and, using some Excel magic, showed us numbers like “number of defects in the web application” and “average number of days a critical bug sat unassigned”. This is good stuff; seeing hard data about what we’re doing lets us focus on inefficiency. Since we release software every month (and launch new clients almost as frequently), this is a huge help in making each iteration better than the previous.

While Anh’s hand-jammed report was awesome, manually connecting SQLite and Excel is no fun for anyone. I whipped up a basic Ruby application using Sinatra to automate most of what we’d like to see. We can now see how long critical defects site open and unfixed, as well as how many overall defects we’re fixing and even the root source of our defects (e.g. did we get the wrong requirements, or just mess up implementing them?)

We still need to show trends, however. “Classic” thinking in software process improvement says to do that by dividing everything by lines of code; if we had 10 defects per 1,000 lines of code this month, but only 8 the following, we’re doing better. This is not a great measure for us, since we are changing and enhancing our product (as opposed to writing fresh code each time), so we decided to normalize our measurements by the story points we assigned to the iteration. This way, we can see charts like the one below (which says we are doing better each time):

This information is dead simple to collect; the burden on developers and testers when writing defects is almost nothing. With only a priority, severity, and “source of defect”, we can collect a lot of advanced information about our development process — and make it better iteration by iteration.