Clicking around google’s well-documented “get started” guides, it looked like they haven’t made their client JARs available in any MVN repos anywhere.  There is a (not-google-sponsored) source forge project that points to a sonatype repo, but instead of adding another 3rd party repo to our small but growing list, I figured it’d be easier for small proof of concept purposes to just manually install the google JARs in OPOWER’s repo (thereby making them accessible to all our developer’s boxes).

It quickly became clear that the number of JARs google ships with makes it faster and less error prone to script the mvn repo installation than doing manual command line copy & pasting.  Lo, my gift to the world:

• Grab latest JARs from the “gdata-java” link here: http://code.google.com/p/gdata-java-client/downloads/list
• Unzip on the server hosting your company’s mvn repo
• cd down in to ./gdata/java/lib
• Make a file ‘installall.pl’ in that lib directory:

#!/usr/bin/perl -w

my @jars = `ls *.jar`;
chomp(@jars);
foreach my $jar (@jars) {
  if($jar =~ /(.*)-(\d\.\d)\.jar/) {
    my $cmd = "./install.pl --artifactId $1 --version $2 --jar $jar";
    open(CMD, "$cmd|") or die "Could not execute $cmd $!";
    while(<CMD>) {
       print $_;
    }
   close(CMD);
  }
}

• Still in that directory, make another file ‘install.pl’:

#!/usr/bin/perl -w

use strict;
use warnings;
use Getopt::Long;

sub usage();

my $artifactId;
my $version;
my $jar;
GetOptions('artifactId=s' => \$artifactId ,
           'version=s' => \$version,
           'jar' => \$jar);
die usage() unless ($artifactId && $version && $jar);

my $cmd = "mvn deploy:deploy-file " .
      "-DgroupId=com.google.gdata " .
      "-DartifactId=$artifactId " .
      "-Dversion=$version " .
      "-Dfile=$jar " .
      "-Dpackaging=jar " .
      "-DgeneratePom=true " .
      "-Durl=file:///opt/mvn_repo " .  <---  change this for your env
      "-Drepository=opower_local";  <--- change this for your env
print "executing command = $cmd\n";
$cmd = "date";
open(CMD, "$cmd|") or die "Could not exec $cmd $!";
while(<CMD>) {
    print $_;
}
close(CMD);

sub usage() {
    print "Usage: ./install.pl [--artifactId artifactId] [--version version] [--jar jarfile]\n";
    print "Example: ./install.pl --artifactId gdata-webmastertools --version 1.0 --jar gdata-webmastertools-1.0.jar\n";
    exit 1;
}

• Don’t forget to chmod 755 *.pl
• Then just run ./installall.pl and all those google JARs should get installed in your repo

That assumes the “mvn” executable was in your path and that you run the scripts from within the same directory as all the JARs, but it’s easily modified for whatever your situation may be.

Note that relative to ./lib there are 2 jars in ../deps/ that you should also install in your repo because the google JARs to avoid ClassNotFoundExceptions in some runtime code paths.

Once in, you should be able to boot strap a mvn client project against a Google Data source with a pom not too dissimilar from:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <artifactId>foo</artifactId>
  <packaging>jar</packaging>
  <version>1.0.0-SNAPSHOT</version>
  <name>Google Client Example</name>

  <scm>
    <developerConnection>foo</developerConnection>
  </scm>

  <dependencies>
    <dependency>
        <groupId>com.google.gdata</groupId>
        <artifactId>gdata-core</artifactId>
        <version>1.0</version>
    </dependency>
    <dependency>
        <groupId>com.google.gdata</groupId>
        <artifactId>gdata-analytics</artifactId>
        <version>2.1</version>
    </dependency>
    <dependency>
        <groupId>com.google.gdata</groupId>
        <artifactId>gdata-client</artifactId>
        <version>1.0</version>
    </dependency>
    <dependency>
        <groupId>com.google</groupId>
        <artifactId>google-collect</artifactId>
        <version>1.0-rc1</version>
    </dependency>
  </dependencies>
</project>

Goodhart’s Law: Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes

Here at OPOWER, developers and product managers don’t deal with KPIs on a day-to-day basis….I think it’s mostly a management-level “let’s starting capturing some numbers over several months and see if there’s anything out-of-whack” kind of thing.

In terms of development KPIs, some of the things we thought it might be interesting to collect included:

• story points delivered per iteration
• % of story points (aggregate) done in an iteration that were planned at the beginning
• % QA coverage of production (automated)
• etc.

The “story points delivered per iteration” is, I think, precisely the kind of thing Goodhart was talking about when he made up his law.  That said, we are a business and we are expected to work and be productive, so it isn’t exactly a satisfying for a VP or CEO to hear from their dev team “sorry, you can’t measure us because we’ll just skew the measurement number to please you.”  So what’s management to do with their IT cabal?

Assuming you could normalize what story points mean across different teams, and assuming you can account for “point inflation” and assuming you accurately tracked vacations and network outages and late requirements and all the other stuff that goes along with uncertainty in exactly how much gets delivered in an iteration, must you also assume that the team gradually skews the number to meet the goal?  What if the goal was kept secret?  What if the team didn’t know they were being measured?  That’s hardly a way to foster a healthy relationship between management and product development.

One way around this dilemma is to find some totally objective metric in the software engineering process.  In other words, if you could apply KPIs to “number of sprockets coming off the assembly line such that each sprocket is within .1% tolerance,” what is a similar kind of thing in our iteration process we could measure?  I.e., one that doesn’t easily fall prey to inflation or manipulation?  If you can find something like that, lemme know.  Until then, I won’t be holding my breath.

Another possible option would be to go ahead and publicly measure highly subjective KPIs anyway and ask the measurees (us) to be as objective as possible when measuring.  I.e., fly in the face of Goodhart’s Law.  After all, it’s just a Law.  Like De Morgan’s Law, except maybe with a bit more wiggle room.

But first, let’s talk about defensive coding and how we do it. For example:

/**
 * frobs the bar and baz.
 * @param bar the bar to frob; may not be null
 * @param baz the baz to from; must be positive
 */
public String frob(String bar, int baz) {
  if (bar.length() > baz) {
    return bar.substring(baz);
  }
  else {
    return bar;
  }
}

If the contract is violated by the caller, we will get a NullPointerException or some other StringIndexOutOfBoundsException. Not too helpful. Instead, we use an internal validator helper class to provide useful messages. This is crucial for properly understanding bugs that occur in production; there’s nothing worse than coming into work to find a NullPointerException in your inbox and no clue what went wrong.

/**
 * frobs the bar and baz.
 * @param bar the bar to frob; may not be null
 * @param baz the baz to from; must be positive
 */
public String frob(String bar, int baz) {
  Validate.notNull(bar,"bar must not be null to frob");
  Validate.isTrue(baz > 0,"baz must be positive");
  if (bar.length() > baz) {
    return bar.substring(baz);
  }
  else {
    return bar;
  }
}

Validate.* methods essentially throw IllegalArgumentException if what they are checking for fails.
We use this pattern extensively, even in constructors of simple data-structure style objects. This (plus keeping this things immutable) allows users of these objects to safely rely upon the get methods behaving properly. I had need of a class to hold both a Person (representing a website user) and a Customer (representing a utility-company customer). This class (called, naturally, PersonAndCustomer), takes both objects in its constructor and requires that they are both non-null. As a further sanity check, I wanted to make sure that the Person’s customerId matched the id of the Customer that was being passed in (i.e. that they represented the same actual human in the real world):

public PersonAndCustomer(Person p, Customer c) {
  Validate.notNull(p,"Person may not be null");
  Validate.notNull(c,"Customer may not be null");
  Validate.isTrue(p.getCustomerId() == c.getId(),
    "Person %d's customer id %d didn't match passed-in customer's id %d",
    p.getId(),
    p.getCustomerId(),
    c.getId());
  ...
}

Looks pretty inocuous, right? Well, because these objects are Hibernate-managed, all of our ids are Long and not long. So, my isTrue validation is really checking that the person’s customerId object is the same object as the customer’s id object. What’s worse, in several months of development and production deployment, these two objects happened to be the same.

Until this week; we were testing our application for a new client and I got the error posted above (“Person 1001’s customer id 109032 didn’t match the passed-in customer’s id of 109032″). It seems that much like how the JVM will re-use string objects, it also will sometimes re-use boxed objects as well. But not always. The fix for this is obvious, but I wanted to make sure I could actually recreate this situation. So, I created a test that gave both the Person and Customer the same value for the customer id, but as different objects. and verified that the class could still be constructed. Sure enough, the test failed. The fix:

public PersonAndCustomer(Person p, Customer c) {
  Validate.notNull(p,"Person may not be null");
  Validate.notNull(c,"Customer may not be null");
  Validate.isTrue(p.getCustomerId().equals(c.getId()),
    "Person %d's customer id %d didn't match passed-in customer's id %d",
    p.getId(),
    p.getCustomerId(),
    c.getId());
  ...
}

What’s interesting is that if one end of the == is a primitive, the JVM will unbox the other one and the test succeeds:

Long l1 = new Long(45L);
Long l2 = new Long(45L);

System.out.println(l1 == l2);      // false
System.out.println(l1.equals(l2)); // true
System.out.println(l1 == 45L);     // true!

The moral of the story is to always know what you are comparing. Might even be best to always use .equals() unless the compiler complains.

There have been a couple articles recently about the effects of sitting in a chair all day that pushed me to action.  This article from business week was particularly eye opening.

OPOWER is super cool about empowering developers to be more productive/comfortable/happy/etc so getting approval for expensing some generic Ikea furniture was as easy as a head nod.  A couple weekends ago, I drove down to Ikea and picked up an Utby desk for about $100 and an Ekby Viktor shelf with Capita legs for another $20 or so.  I didn’t think the desk by itself was going to be quite at the recommended elbow height, so I swung by Home Depot and got 4 cinder blocks too.

I ended up putting a stack of paper between my monitor and the shelf to really get it up at eye level and I’m pretty happy with the way it turned out:

Dave’s setup is very similar, but he seems to like focusing down on his single screen whereas I like having a couple monitors at different focus points where I can physically distinguish tasks (documentation, music, images on my laptop off to the side and development tasks on the main monitor straight ahead).

Dave at his standing desk

It’s been a couple weeks since I switched and I can draw the following conclusions so far:

• Your feet hurt like a sonofabitch at the end of 8, 9, 10 hours if you’re not used to standing that whole time
• A little bar stool or some off-the-floor step helps a lot in shifting your weight around so you’re not always bearing down on both heels
• My back feels a ton better
• I feel more productive (though I don’t know empirically if I actually am) because (a) I feel really focused on the monitor right in front of me — more so than when sitting down and (b) I’m a little more self-conscious of my screen being visible to everyone walking by, so I suspect I’ve got a higher percentage of work-related stuff on my screen at all times than I did before.

To the disgust of myself and my wife, I collected shavings for approximately 4 months in a plastic cream cheese jar labeled “Not Cream Cheese.”

This last weekend, I took a trip to AC Moore Arts & Crafts and bought some clay.  I fashioned a mold into that clay of a mustache that would dominate all other mustaches.

Into that mold, I poured elmer’s glue, hair shavings, 14 gauge electrical copper wiring, more hair, more glue, more hair.

This morning, the plan was to simply glue the assembled mustache on to my real mustache.  Unfortunately, the plan involved entirely too much super glue and the assembled mustache pieces proved to be entirely too heavy.  My previous experience with krazy-glue-on-skin is probably like most people’s…stick your thumb and index finger together.  Thanks to callouses, a lack of hair on your fingertips and the relative strength of your digits, it’s not that big of a deal.  I also don’t remember the glue burning like lava on contact.

A split second before squirting a liberal does of krazy glue on the mustache I wondered: “is this really a good idea?”  But not wanting to pysch myself out, I quickly brought the assembled mustache up to my lip and pressed down hard.

The first thing I remember thinking is that this was going to look really good.  Like, maybe champion-quality entry.

That thought was quickly replaced with a rising alarm about the rapidly increasing burning sensation I was experiencing between the mustache and my lip.  That alarm was then replaced with the growing realization that I was inhaling quite a bit of vapor and that my throat was starting to burn.

Not-quite-panicked, but not totally calm either, I gently tried to remove the mustache and got as far as pulling my lip about an inch off my face before giving up.  Over the next 10 minutes I was able to saw, snip and shave the ’stache down with a combination of electric shaver, electrical pliers, scissors and razor blade.  By that time, of course, the fumes had dissipated and the burning was just a low heat and my grand mustache plans were wrecked.

The redness of my eyes is due to the fumes and pain.

There’s always 2011…

We here at OPOWER use Alfresco Community 3.2 as a glorified “XML editing tool” that was easy to setup, configure and hand over to the business to allow them to contribute XML-based content that our applications can consume. As a F/LOSS solution, it’s fit our needs pretty well and is generally a low-maintenance piece of our infrastructure.

I attribute my “low maintenance” characterization of an Alfresco installation mostly to the patience of my small user base, who are very accommodating to the quirks and limitations of the installed (and neglected) instance.

The 3.3 lunch and learn was intro’d by Chris from SiteWorx who surveyed the CMS landscape and equated the emerging CMIS spec as “content’s ODBC” which I thought to be a quite fitting analogy.

Richard Im from Alfresco then spoke for about 45 minutes about the new features Alfresco 3.3 contained:

WCM Improvements

• new web-based editor, reminiscent of Interwoven’s in-context editor from the 6.5/7.0 era
• new bi-directional deployment capabilities (including addition of “delivery server” infrastructure possibilities like several other CMS offerings)
• “Straight through” publishing without workflows
• google-like search on metadata attributes

Share

• Evolving share into more of a bit-for-bit replacement of the “traditional” UI (which Richard referred to as “DM”… “document manager?”)
• Permission control a la DM
• Rules and Actions just like in DM
• Data Lists (like Share Point’s project lists)

I asked the question: “what’s the medium-term road map for DM, if Alfresco seems intent on mirroring all(?) functionality in the traditional UI via the /share context?”  Richard wouldn’t definitively say that the traditional app is going to be deprecated any time soon, but it certainly appears that way…

“Integration” improvements

I think this suite of improvements is geared towards the enterprise crowd:

• Full CMIS 1.0 spec implemented as of 3.3 (the only open source implementation known to do so)
• ACLs
• Change Logs
• Additional SOA/REST services and support
• Ability to save off google docs in alfresco for management (!)

My big take-aways from the lunch and Q&A session was that Alfresco’s 3.3 feature list and 3.4 direction seem to put them on more of a competitive footing with monster ECM vendors like Interwoven.  It’s maybe an odd direction for an open source project to steer because while Alfresco certainly employs their own development team, I’m not sure how much community interest there is in contributing to some of the more mundane enterprise ECM concerns that need to be built or integrated.  On the other hand, Alfresco, Inc. does survive with enterprise support contracts that smaller shops like mine are unlikely to ever pay top-dollar for, so maybe it’s a long term survival strategy.

Long term, I’m interested in exploring architectures available to us with a SOA-enabled CMIS layer and eventually ditch the custom-DAO layer we’re running right now against Alfresco-exported XML flat files.  Even if that never happens, it’s exciting to see the kinds of complexity that open source projects are able to attack.

We have lots of OSes around the office here – Ubuntu, Windows, and OSX. Of those, Ubuntu and Windows natively provide a way to lock your desktop with a keystroke (Win+L or Ctrl+Alt+Del, Enter on Windows and Ctrl+Alt+L on Ubuntu). Strangely, Mac gives no easy way to lock your desktop.

We’ve been using HotCorners for the screen saver, but who wants to touch the mouse if you don’t have to (especially when you’re racing to grab a free lunch)?

Enter Quicksilver, everyone’s favorite application interface. (There are all sorts of things you can do with Quicksilver and its various plug-ins, but you can look that up yourself.) To set up a keyboard shortcut with Quicksilver to lock your screen (hat tip to Bryan Helmkamp):

1. Create a symbolic link to your screen saver application:
   $ sudo ln -s /System/Library/Frameworks/ScreenSaver.framework/Versions/Current/Resources/ScreenSaverEngine.app/ /Applications/Screen\ Saver.app
2. Quicksilver -> Triggers -> Custom Triggers
3. Add a Hotkey
4. Find the ScreenSaverEngine by typing and press enter
5. Double-click the cell in the Trigger column
6. Choose your hotkey (I like Ctrl+Cmd+\ since it’s almost Ctrl+Alt+Del
7. Close the window (you may also need to restart QS

Now set up your System Prefs in Security -> General to require a password immediately after screen saver begins and you’re set.

Sadly, you need to do this or your screen saver will awake as soon as you release your QS shortcut. This even happens if you set QS to launch screen saver “On Release”.

Anyway, after that, lock screen is just a keystroke away.

My first thought was: there’s lots of blocking reading and writing from the database, so adding some threads should speed things up. While one thread was analyzing Bob’s energy data, another could be fetching Mary’s, while another could be updating Joe’s meta-data with the results. Or so I thought.

The more threads I added, the slower the entire thing became. It turned out that the fastest implementation was a single-threaded one. But why? It all has to do with the diminishing returns one gets from scaling out.

If you think of a task as having a serial component, which cannot be broken up concurrently, and then multiple tasks which can be done concurrently, we can analyze the returns we get by increasing the number of available processors (threads, in my case). This is Amdahl’s Law and is exemplified by the following equation (where “x” is the number of processors or threads, and “s” is the percentage of your overall task that must be serialized; “y” is the increase in speed you will see from scaling).

When you graph this, it’s pretty obvious that there are diminishing returns to adding more threads/processors (the graph below assumes that 90% of the overall job can be done concurrently). As we add threads, we get less and less of a gain in speed.
But there’s still a gain to get, so what happened to me?

Amdahl’s law is actually pretty optimistic. It doesn’t account for the overhead required to synchronize the shared data. If we account for this, with a new value “k” ( the percentage penalty for maintaining consistency), we see that increasing our processors/threads actually starts to hurt us (this equation is in red)!

In my case, almost all of this synchronization is happening within the database; it turns out that almost all the time my process needs is accessing data from the database. So, I dialed it down to only one thread, and we should be done early next week.