Clicking around google’s well-documented “get started” guides, it looked like they haven’t made their client JARs available in any MVN repos anywhere.  There is a (not-google-sponsored) source forge project that points to a sonatype repo, but instead of adding another 3rd party repo to our small but growing list, I figured it’d be easier for small proof of concept purposes to just manually install the google JARs in OPOWER’s repo (thereby making them accessible to all our developer’s boxes).

It quickly became clear that the number of JARs google ships with makes it faster and less error prone to script the mvn repo installation than doing manual command line copy & pasting.  Lo, my gift to the world:

• Grab latest JARs from the “gdata-java” link here: http://code.google.com/p/gdata-java-client/downloads/list
• Unzip on the server hosting your company’s mvn repo
• cd down in to ./gdata/java/lib
• Make a file ‘installall.pl’ in that lib directory:

#!/usr/bin/perl -w

my @jars = `ls *.jar`;
chomp(@jars);
foreach my $jar (@jars) {
  if($jar =~ /(.*)-(\d\.\d)\.jar/) {
    my $cmd = "./install.pl --artifactId $1 --version $2 --jar $jar";
    open(CMD, "$cmd|") or die "Could not execute $cmd $!";
    while(<CMD>) {
       print $_;
    }
   close(CMD);
  }
}

• Still in that directory, make another file ‘install.pl’:

#!/usr/bin/perl -w

use strict;
use warnings;
use Getopt::Long;

sub usage();

my $artifactId;
my $version;
my $jar;
GetOptions('artifactId=s' => \$artifactId ,
           'version=s' => \$version,
           'jar' => \$jar);
die usage() unless ($artifactId && $version && $jar);

my $cmd = "mvn deploy:deploy-file " .
      "-DgroupId=com.google.gdata " .
      "-DartifactId=$artifactId " .
      "-Dversion=$version " .
      "-Dfile=$jar " .
      "-Dpackaging=jar " .
      "-DgeneratePom=true " .
      "-Durl=file:///opt/mvn_repo " .  <---  change this for your env
      "-Drepository=opower_local";  <--- change this for your env
print "executing command = $cmd\n";
$cmd = "date";
open(CMD, "$cmd|") or die "Could not exec $cmd $!";
while(<CMD>) {
    print $_;
}
close(CMD);

sub usage() {
    print "Usage: ./install.pl [--artifactId artifactId] [--version version] [--jar jarfile]\n";
    print "Example: ./install.pl --artifactId gdata-webmastertools --version 1.0 --jar gdata-webmastertools-1.0.jar\n";
    exit 1;
}

• Don’t forget to chmod 755 *.pl
• Then just run ./installall.pl and all those google JARs should get installed in your repo

That assumes the “mvn” executable was in your path and that you run the scripts from within the same directory as all the JARs, but it’s easily modified for whatever your situation may be.

Note that relative to ./lib there are 2 jars in ../deps/ that you should also install in your repo because the google JARs to avoid ClassNotFoundExceptions in some runtime code paths.

Once in, you should be able to boot strap a mvn client project against a Google Data source with a pom not too dissimilar from:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <artifactId>foo</artifactId>
  <packaging>jar</packaging>
  <version>1.0.0-SNAPSHOT</version>
  <name>Google Client Example</name>

  <scm>
    <developerConnection>foo</developerConnection>
  </scm>

  <dependencies>
    <dependency>
        <groupId>com.google.gdata</groupId>
        <artifactId>gdata-core</artifactId>
        <version>1.0</version>
    </dependency>
    <dependency>
        <groupId>com.google.gdata</groupId>
        <artifactId>gdata-analytics</artifactId>
        <version>2.1</version>
    </dependency>
    <dependency>
        <groupId>com.google.gdata</groupId>
        <artifactId>gdata-client</artifactId>
        <version>1.0</version>
    </dependency>
    <dependency>
        <groupId>com.google</groupId>
        <artifactId>google-collect</artifactId>
        <version>1.0-rc1</version>
    </dependency>
  </dependencies>
</project>

Goodhart’s Law: Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes

Here at OPOWER, developers and product managers don’t deal with KPIs on a day-to-day basis….I think it’s mostly a management-level “let’s starting capturing some numbers over several months and see if there’s anything out-of-whack” kind of thing.

In terms of development KPIs, some of the things we thought it might be interesting to collect included:

• story points delivered per iteration
• % of story points (aggregate) done in an iteration that were planned at the beginning
• % QA coverage of production (automated)
• etc.

The “story points delivered per iteration” is, I think, precisely the kind of thing Goodhart was talking about when he made up his law.  That said, we are a business and we are expected to work and be productive, so it isn’t exactly a satisfying for a VP or CEO to hear from their dev team “sorry, you can’t measure us because we’ll just skew the measurement number to please you.”  So what’s management to do with their IT cabal?

Assuming you could normalize what story points mean across different teams, and assuming you can account for “point inflation” and assuming you accurately tracked vacations and network outages and late requirements and all the other stuff that goes along with uncertainty in exactly how much gets delivered in an iteration, must you also assume that the team gradually skews the number to meet the goal?  What if the goal was kept secret?  What if the team didn’t know they were being measured?  That’s hardly a way to foster a healthy relationship between management and product development.

One way around this dilemma is to find some totally objective metric in the software engineering process.  In other words, if you could apply KPIs to “number of sprockets coming off the assembly line such that each sprocket is within .1% tolerance,” what is a similar kind of thing in our iteration process we could measure?  I.e., one that doesn’t easily fall prey to inflation or manipulation?  If you can find something like that, lemme know.  Until then, I won’t be holding my breath.

Another possible option would be to go ahead and publicly measure highly subjective KPIs anyway and ask the measurees (us) to be as objective as possible when measuring.  I.e., fly in the face of Goodhart’s Law.  After all, it’s just a Law.  Like De Morgan’s Law, except maybe with a bit more wiggle room.

There have been a couple articles recently about the effects of sitting in a chair all day that pushed me to action.  This article from business week was particularly eye opening.

OPOWER is super cool about empowering developers to be more productive/comfortable/happy/etc so getting approval for expensing some generic Ikea furniture was as easy as a head nod.  A couple weekends ago, I drove down to Ikea and picked up an Utby desk for about $100 and an Ekby Viktor shelf with Capita legs for another $20 or so.  I didn’t think the desk by itself was going to be quite at the recommended elbow height, so I swung by Home Depot and got 4 cinder blocks too.

I ended up putting a stack of paper between my monitor and the shelf to really get it up at eye level and I’m pretty happy with the way it turned out:

Dave’s setup is very similar, but he seems to like focusing down on his single screen whereas I like having a couple monitors at different focus points where I can physically distinguish tasks (documentation, music, images on my laptop off to the side and development tasks on the main monitor straight ahead).

Dave at his standing desk

It’s been a couple weeks since I switched and I can draw the following conclusions so far:

• Your feet hurt like a sonofabitch at the end of 8, 9, 10 hours if you’re not used to standing that whole time
• A little bar stool or some off-the-floor step helps a lot in shifting your weight around so you’re not always bearing down on both heels
• My back feels a ton better
• I feel more productive (though I don’t know empirically if I actually am) because (a) I feel really focused on the monitor right in front of me — more so than when sitting down and (b) I’m a little more self-conscious of my screen being visible to everyone walking by, so I suspect I’ve got a higher percentage of work-related stuff on my screen at all times than I did before.

• Iteration N, Week 1 = Design for iteration N and 2nd week of QA for iteration N-1
• Iteration N, Week 2 = Development
• Iteration N, Week 3 = Development
• Iteration N, Week 4 = 1st week of QA for iteration N
• Iteration N, Week 5 / Iteration N +1, Week 1 = 2nd week of QA for iteration N, design for iteration N + 1

We started to notice a couple things that have caused us to try out a new schedule in our upcoming 6th iteration.

First, we realized that we left very little time for fit to hit the shan.  If we had a tough QA cycle, we didn’t design enough.  If we didn’t design enough, we underestimated how much development would be needed (i.e. too many story points).  If we had too many story points, we’d go in to QA late.  And the cycle repeats.

Note that many of you Agile purists out there will cringe at the above description, especially this sentence: “If we had too many story points, we’d go in to QA late.”

I know.  We cringed too.

We’re committing to a couple new explicit practices going in to our next iterations:

1. More formal velocity check-ins.  If less than 50% of our story points are closed by COB of the first Friday, we should have a come-to-jeebus meeting on Monday morning.
2. We absolutely need to respect the impact that dedicated design time has on the effectiveness of everything down-stream.  It’s too easy to say “Agile” as a synonym for “code by the seat of your pants,” but that’s not what it’s about and we know that.  We just need to get better at dedicating time to it.

So with those ideas in mind, we’re moving to something like the following:

It’s a five-week iteration, with 2 solid weeks built in for QA and 1 dedicated week of design.

I think if we can get to a point where our regression and automation testing is good enough (and we develop enough confidence our automated tests), that we may be able to get back to a 4 week cycle.  In the mean time, we’re going to see if we can actually go faster by slowing down.

Stay tuned!

Well, we here at OPOWER loved our Duck Hunt, so we’ve been attempting to use “Agile” for the last 5 months or so. Agile means a lot of things to a lot of people, so I’ll spell out what it means for us and then reflect on some of the pros, cons, and room-for-improvement in our practice.

• <cringe>Embrace change</cringe> (duh)…but especially in this startup environment, we need to remember to do no more than is necessary, because any “extra mile” functionality is likely to be totally unused as priorities and direction changes.
• Be scientific about development…measure, measure measure.  We track velocities, average-open-days-per-bug-by-severity, etc.
• “Individuals and interactions over processes and tools”…we self-organize, aren’t afraid to try new combinations of responsibilities and assignments, provide regular feedback to each other and we don’t dictate a development environment or technology stack.

The best thing about the technical side of this company is that the entire development team and project management team is on-board with the idea of rapid iterations, continuous improvement and a focus on technical excellent and “maximizing the amount of work not done.”

This manifests itself in several practices that I’d chalk up in the “pro” column:

• Light weight documentation…use wikis and person-to-person contact (recorded on bug tickets for posterity) to flush out designs and requirement.
• Continuous integration and continuous improvement (we just installed Sonar last week…we’ll blog about its effectiveness later)
• Regular code reviews, 360 reviews, and “how did we collectively perform this iteration” meetings
• Few formal meetings, many informal, impromptu “drive bys”

But the best thing about our Agile environment right now is that it is a work-in-progress “process framework” that repeats quickly enough for us to remember what to improve.

What do I mean by that?

It’s my impression that on the longer, more drawn-out software engagements (say…6 or 12 months), you can make a series of mistakes in the design phase, the build phase, the qa phase, etc. and then by the time you’re on the next project you repeat all those mistakes again because it’s been 6 or 12 months since you last learned your lesson.  Instead of concrete lessons, you develop a career full of “gut instincts.”

We’re still honing in on the right amount of measurement and process for OPOWER, but because we’re all on board with the idea of quick-hit iterations with a continuous focus on improvement, I have high hopes for our ability to become (stay?) a top-notch software engineering organization.

There’s lots of room for improvement, that I think I’ll save for another post on another day.  We could definitely stand to “communicate out” more– with the product team and with our operations team.  We need to do a better job of making sure our iterations are lining up with the corporate strategy.  We could improve on the whole “customer collaboration over contract negotiation” part of the manifesto.  But you can only squeeze that trigger so fast, right?

You cruise in to the office at 8:45, get your coffee, check the reddits, and ignore that email from your parents asking if their computer picked up a virus because McAfee won’t stop popping up a tooltray icon helpfully informing them that they’re totally vulnerable. Right around 9:15, you get motivated to tackle that 2 point user story for the current iteration.

You head over to the wiki, check out the specs for the requirement, find out what the Trac ticket number is and see if there are any comments with last minute advice or dependencies. None? Good. . . let’s get started!

Oh, huh. . . it’s 9:40. There’s that all-hands meeting at 10 o’clock. I could get started, but I’m not really going to get anything done in 20 minutes really….and that link to the keyboard-cat-three-wolf-t-shirt did look interesting.

The 10 o’clock meeting runs a little long, and then there’s the lunch train, and suddenly it’s the afternoon and you haven’t been in The Zone once.

The Zone

We’ve all been in The Zone – that magical place where time appears to stop and you rock out an insane amount of code in what seems like 10 minutes, but when you look up from your keyboard the sun is down and the cleaning crew is giving you a weird look because you didn’t realize you were trying to sing along to Sigur Ros.

Trying to force yourself into The Zone is like trying to force yourself to see through those stereograms (hint: it’s always a Schooner). And as hard as it is to get in The Zone, it’s annoying easily to get out of The Zone.

There are a lot of recommendations out there to help create “zone-friendly” work places including:

• separate offices
• ambient music
• leaving something intentionally broken the night before to give you that excuse to dive in the next morning
• shift your tasks to the upper right
• Paul Graham’s thoughts on the matter
• etc. etc.

We were wrapping up our all-hands meeting yesterday when the ever-perceptive Dave noted that our weekly meeting schedules were really not all that Zone-Friendly. With a couple of exceptions, we had standing meetings along the following schedule:

And keep in mind, that’s just the standing meetings, so it’s the bare-minimum one would expect. Now let’s add some “procrastination shading” to that calendar to demonstrate the actual standing meeting impact:

If we shuffled some meetings around (especially around the meat of the middle of the week), I think we could end up “finding” 2 or 3 hours a week more conducive to The Zone:

Of course, this is an idealized example, but it’s something we’re thinking about and playing with.

What do you guys out there in development land do? Any recommendations of particular effectiveness?

If you throw an exception about some connection being refused, you’d better damn well put what connection you were attempting to make in the exception message.

What inspired this law?  Thanks for asking.

Check out this snippet in the middle of an 80 line stack dump:

2009-08-25 17:23:42.373::WARN:  Nested in org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'coreAutopatcher' defined in class path resource [config/migrationContext.xml]: Invocation of init method failed; nested exception is com.tacitknowledge.util.migration.MigrationException: Error applying patches:
com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1319)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)

“A ResourcePool could not acquire a resource from its primary factory or source”. Awesome. Sooo helpful. /wristsemoragegrowl

Also, a WARN? Seriously? You’re a connection pool. Do you think not connecting to your underlying datasource warrants maybe something as stern as an ERROR? Or, heaven forbid, a FATAL?

I’m sure I could have turned up logging somewhere, but I noticed Spring’s DelegatingDataSource bean standing out from among the stack lines and set a break point on its called method to figure out exactly which of my many connections was unable to be resolved:

There’s the culprit! localhost:5455. You know, whoever the idiot is who runs that localhost server needs to get his act together. I should call the guys from YouTube and have them do a Maury Povich-style video expose on why he can’t keep any of his connections stable.

svn co http://domain.com/svnroot/project/trunk project

That way, when I go to the project directory, I do not have to then go into the trunk directory to get to my files. Saving an extra directory level is not the best benefit though. When I set up my IDE (yes, I use an IDE), I put the IDE files in that one directory and do not have to recreate them for each branch or tag that I might check out. I essentially treat the “trunk/branches/tags” part of the directory as pure metadata, which makes perfect sense to me.

When I switched over to this system, there was one big drawback. I could not easily tell where I am just by looking at my prompt. I had to run svn info | grep URL to see if I was on a branch. So instead of letting my prompt handicap me, I decided to get smart about my prompt. Using bash, my prompt used to look like this:

jeff:/opt/pose/main/report/src/main/java

but now it looks like this (supercharged with Subversion metadata):

jeff:/opt/pose/main/report/src/main/java [SVN: /main/report/trunk]

I’m no bash expert, so I dug around on the web and learned a bit about how PROMPT_COMMAND works. Here is what I have in my .bash_profile now:


function spwd ()
{
stat .svn > /dev/null 2>&1;
if [ "$?" == "0" ]; then
SURL=`svn info | grep URL | perl -pe 's/URL: (.*)/\1/'`;
if [ `echo ${SURL} | grep -E "branches|tags"` ]; then
SVER=`echo ${SURL} | perl -pe 's{.*/(branches|tags)/(.*)}{\1/\2}' | cut -d/ -f1-2`;
SPTH=`echo ${SURL} | perl -pe 's{.*svnroot/(.*)/(branches|tags)/.*}{/\1}'`;
SPWD="${SPTH}/${SVER}";
else
SPWD=`echo ${SURL} | perl -pe 's{.*svnroot/(.*)/trunk(.*)}{/\1/trunk}'`;
fi;
export PS1="\u:\w [SVN: $SPWD]\n$ ";
else
export PS1="\u:\w $ ";
fi
}
export PROMPT_COMMAND=spwd


It may not be the most efficient way to do it, but it sure works well and lets me know in an instant what trunk, branch or tag I happen to be working on at that very moment.

Still, the “everyone else” paint brush was wearing thin.  In addition to the “web team”, we had “the scale team” (a.k.a “Ops R&D”) and “the AMI team” and “the core team” and “the report team”. . .but most of those were “teams” averaged size 1 and their membership varied from iteration-to-iteration.

Sick of referring to ourselves with a hand-wave and a “you-know. . .the non-web-team,” I plugged our names in to an acronym engine and we found the perfect name for ourselves:

The Cheated Frogs of Vim

While I put in my fair share of vi time in previous development gigs, I’m a little too fond of modern debuggers to go whole-hog with Vim as my “IDE.”  Dave, on the other hand, is pretty hard-core with his Vim skills, so our new team name may have some sticking power.

Our wizard graphics guy even cooked up a logo for us:

New dev team logo!