Anatomy of a P.I.R.A.T.E. Build Monitor

As I have blogged before, I have a slight obsession with build monitors. That obsession has probably peaked with the creation of the P.I.R.A.T.E. or the Peripheral for Information Radiation, Audio and Telemetry Enabled. What follows is a description of the main components. I’ll follow up with subsequent blogs on what I learned about group dynamics and some of negative the impacts of scaled Agile has on the individuals it is imposed upon. All very unscientific but it started me on a path of self-study into conflict in organizations and the project I was on. Sounds scary so let’s start with the more playful side of the P.I.R.A.T.E.

IMG_5140

The P.I.R.A.T.E. in his glory days.

On the first project in which he made an appearance, he started as a simple build monitor using the technology of his predecessors:

  • Arduino with a few tri-color LEDs attached. The colors and be controlled to get Red, Green and Blue. Yellow is hard with the LEDs to get so I switched yellow for purple on broken unit tests.
  • Puppy Linux based PC attached to the Arduino using the USB connection. I built it before Raspberry Pis became the rage. The colors could then be changed to match whatever the build conditions were.
  • I covered the PC case with a pirate ship cover made from wood.
  • I made a mainsail from a canvas from the local art store. I used a T shirt inkjet transfer to get a cartoon one of the developers created with our mascot Thunder Penguin (more obscure story, that you had to be there for).
  • The P.I.R.A.T.E. Captain was an inverted 8 inch ceiling bathroom light fitting. I managed to find a Halloween decoration kit of a Mister Potato Head Pirate (yes there is even a story behind Mr Potato Head, but either way it worked for this build).
  • I used Processing to develop the display which consisted of a lip-syncing pirate skull. The skull could lip-sync to pre-recorded MP3s. I do a pretty mean pirate voice. This was the main novelty added to the set-up described in my earlier blog. When we added the CI Game I added a realtime display of the top five scorers, the worst scorer (Biggest Bilge Rat) and which build last broke.
  • I added an odd cannon peripheral, as seen in the picture. I’ll have to describe it separately as I used a design technique called TRIZ to develop it that deserves its own blog. For now know the black PVC tubing forms a forms a air flow system driven by a aero-bed fan attached to the rear. The black tubing contains a magazine of ping-pong balls that can be fired out of the silver cannon. The firing is controlled by the Arduino and can be triggered from the Processing app.
  • I hung more pirate paraphernalia around the build monitor. The beer bottle on the right is behind an emergency breakable box (not really). It is a donated home brew called “Ol’ Pirate Piss”. I don’t think I will ever open it.
  • Behind the sail is some old team norms I made to look like an ancient paper declaration of the Articles of the Ship. Despite their bad habits, pirates had a number of self organizing traits we now value on Agile teams.
  • The Portal Royal sign has a story unto itself but is a reference to the Jamaican pirate haven. The Chinese symbol of the one for “Ba”.
IMG_5142

The P.I.R.A.T.E. in his glory days.

The Processing app started out with these basic features:

  • Be able to poll Jenkins for a list of jobs and determine if they were Red, Green (the is a green ball add in that changes the default blue ball) or Yellow.
  • By looking via the Jenkins API at the JSON returned, determine who broke the build and call out their name using a predefined pirate insult. This made use of text-to-speech added to the Processing app.

One his second project, he started to grow some more capabilities.

  • Yahoo Instant Messaging Interface so I could make him speak by sending him text. Good for amusing curious folks who came to look at him. He would say things back.
  • One day one of the remote devs on another team apologized via this IM interface. It was weird to think people thought they were talking to someone, so I thought it should always respond. I added an Eliza library so following the abuse, anybody answering would enter a therapy session with the P.I.R.A.T.E. I ended up recreating the original experiment. More on this as a sign of social disengagement in realtime in another blog.
  • I added a Twitter account @agilepirate which is offline at the time of this blog.
  • We started using Hipchat so I attached the IM response system to it and allowed the P.I.R.A.T.E. to abuse people in his own Hipchat room. I used a Jabber Library to interface to Hipchat. Folks could @pirate in Hipchat and he would speak back.

Probably the biggest addition to the app what integration with the Jenkins CI game addon. This addon would score developers for successful automated builds. The results were available in the JSON after the addon was installed. We took the to the next level when the project’s Yak Shaver took it upon himself to integrate the results of SONAR into the game score. What we did was gamify the code quality it what was a pretty bad legacy code base.

Initially it was not hard to get big scores simply by doing simple things to improve SONAR scores. Over time the easy wins were less easy but the competition continued. The crafting coders cared about quality and it showed in their scores. Those that showed less interest would often get negative scores. The system was not perfect and we had to manually adjust scores when weird things happened with builds. Those weird things were often things we would not have noticed if not for the game and so became the basis for improvements. Here is a summary of the things we learned from the game:

  • Disk space on CI was insufficient and would cause intermittent failure of builds.
  • Mocks service access via Putty/ssh. It is possible to kill the mock services if accessed in a certain way for deployment. Failing services cause builds to then fail.
  • Database use in CI. When the database fails say due to space issues on the service then builds will fail. Consider using mock DB technology to avoid the dependence on infrastructure.
  • When a team added new technology with consulting the architects, then incorrectly configured the Maven POM, the generated classes were scanned and reduced SONAR points. The resulting 5 Whys found the tool change and started a conversation around alignment to standards.
  • Moral Licensing: The behavior that results from the subconscious mind justifying negative behavior. Here the developer would go out and bank points to make them feel better about possibly breaking the build. Not great behavior because it implies practices are not being applied before checking in or lack of confidence. Still improves quality.
  • Social Loafing: On a big team people get lost. No activity implies disconnection or checking out.

I also added plotting of the results to Thingspeak with a graph per team. It sort of acted as a heartbeat for the Scrum Masters to see if their teams were doing something regardless if the score was going positive or negative. If a team “flatlined”, it was an indicator that maybe they needed technical side of the house to check in.

It was an interesting experiment which we eventually shut down as the increasing turnover meant the team’s motivation to improve faded and as various iteration of program management moved from attempting to be a learning community to a  more schedule and cost focused one. The P.I.R.A.T.E. is currently in dry dock and I am moving on to a new company so he may yet again take to the Seven SVNs. Keep your spyglasses trained on the Agile horizons and beware. The seas of code quality are rough and you might be in need of your own P.I.R.A.T.E.

 

Advertisements

About guywinterbotham

An Agile Buccaneer navigating the corporate storm
This entry was posted in Agile Anthropology, Build Monitor, Play. Bookmark the permalink.

Leave a Message in this Bottle

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s