Pragmatic Automation

Using the Firecracker with Macs

Jim Weirich wrote in to say that he has it on good authority that the KeySpan USA-19HS will properly drive a Firecracker. The Keyspan adapter comes with Mac OS X drivers that speak the handshake signals the Firecracker listens for. Huzzah! That means you can remotely control your lava lamps from your Mac.

Enjoy!

Bubble, Bubble, Build's In Trouble

Your software is being automatically built and tested on a schedule. It even sends you an email when the code doesn’t compile or pass its tests. You’re certainly ahead of most projects, but email is just so 90s. Even if you could manage to find those build failure emails amidst all that spam, you’re reading yesterday’s news. Indeed, you may already be ignoring the status of the scheduled build.

The Monitoring chapter of the book offers alternative, in-your-face, worth-getting-up-for-in-the-morning techniques for monitoring scheduled builds. The most popular technique came by way of a story contributed by Alberto Savoia. He describes how his project uses red and green lava lamps to radiate the status of their scheduled build. Better yet, those lamps are controlled using X10 devices such as those used to turn on your household lamps so that you don’t arrive home to a dark house.

Well, as you might imagine, I could hardly wait to build my very own build-monitoring lava lamp kit. And as bonus material for readers of the book, I’ve crafted a bit o’ software that integrates with CruiseControl. So now you too can enjoy red and green bubbles on your project!

Bill of Materials

To get started, you need some automation gear. Think of these gadgets as this year’s essential project accessories:

4-Piece Firecracker Automation System
This kit includes:
- 1 Firecracker Computer Interface
- 1 Transceiver Module
- 1 Lamp Module
- 1 Palm Pad Remote Control
Cost: $39.99

(Props go to the folks at x10.com for supporting this project by supplying me with a complimentary kit. It all fits in a wee box, so I can carry it from project to project.)

With that kit, you can control two lava lamps — one plugged into the transceiver module and the other plugged into the lamp module. You can optionally purchase another appliance module if you want to control two appliances. For example, you might want your build process to turn on a coffee pot when the build fails and then kick start your margarita machine when the build is fixed.
2 lamps, preferably the kind that boil red and green lava
I used the Hot Rock Lite F/X (yellow earth/blue liquid and red earth/purple liquid). Note for legal purposes that these lamps (shown in pictures below) are not LAVA(R) brand motion lamps, but those will work just as well.

Cost: $9.99 each at Target or Walmart
Pragmatic Automation X10 software
It’s an open source Java library that includes the CruiseControl plug-in, an API to make your wildest X10 dreams come true, detailed instructions, and an ever-so-useful collection of tests.

Way down deep, the library uses the Java Communications API to send bits out over the serial port and into the Firecracker Computer Interface. (Linux users will need the RXTX implementation). Michel Dalal’s Java X10 CM17A API library, an implementation of the FireCracker (CM17A) Communications Specification, is used to send out the correct 1s and 0s in response to human-friendly commands. Many thanks to him for doing all the low-level bit twiddling and sharing the goodies with us!

Cost: Free to readers of Pragmatic Project Automation

Assembling the Kit

With that hardware in hand, you’re ready to start the assembly process. The Firecracker Automation System includes instructions written for your average home electronics consumer, so your average computer/network geek should have no trouble. I’ll spare you all the gory details and instead run through a quick visual tutorial of my setup.

Start by plugging the Firecracker Computer Interface into a serial port of your scheduled build machine:

This little gem sends a wireless signal from the computer to the transceiver module. Notice that you don’t lose the serial port. You can plug another serial device into the back of Firecracker Computer Interface.

Next, plug the transceiver module into any wall outlet within your electrical wiring system. Then turn on the lamp you want the build process to light up when the build fails (the red lamp) and plug it into the transceiver module:

See that antennae on the transceiver? The transceiver picks up the RF signal sent by the Firecracker Computer Interface connected to the computer, converts it into an X10 signal, and broadcasts the X10 signal across the electrical wiring system.

Every X10 module is uniquely identified by a house code (A-P) and a unit code (1-16). By default, the transceiver is configured to listen on "A1". So when the Firecracker Computer Interface sends a signal that tells module "A1" to turn on, the device that’s connected to the transceiver—the red lamp—is turned on.

Next, plug the lamp module into a wall outlet and set its house and unit code to "A2". Then turn on the lamp you want the build process to light up when the build passes (the green lamp) and plug it into the lamp module, like so:

When the Firecracker Computer Interface sends a signal instructing the "A2" module to turn on, the transceiver picks up the signal and broadcasts it out over the electrical wires. The lamp module hears the signal and turns on the green lamp.

That’s it for assembly!

At this point it’s a good idea to make sure you can turn these lamps on and off at will using the free software (separate download) that emulates the Palm Pad Remote Control. It sends signals through the Firecracker Computer Interface, so it’s a good sanity check that you have everything hooked up correctly.

Installing the Software

OK, now you want to hook the lava lamps up to your scheduled build process running in CruiseControl. That’s where the Pragmatic Automation X10 software comes in.

The README file describes how to install and test the software in detail. The final step is to register the CruiseControl plug-in that effectively wires up the lamps to indicate the status of each CruiseControl build cycle. Just to demonstrate how easy that is, here’s the XML you need to add to CruiseControl’s config.xml file:

  <plugin name="x10publisher" classname="com.pragauto.X10Publisher"/>

  <publishers>
    <x10publisher port="COM1" passDeviceCode="A2" failDeviceCode="A1" />
  </publishers>

Edit the attributes of the <x10publisher> element as necessary for your serial port and device codes. A complete config.xml file is included in the project as a reference.

Bubbles While You Work

Once you’ve fired up CruiseControl and a build succeeds, you’ll see something like this on the console:

  BUILD SUCCESSFUL
  Total time: 10 seconds
  [cc]Jul-08 16:56:53 Project       - Project dms:  merging accumulated log files
  [cc]Jul-08 16:56:53 Project       - Project dms:  publishing build results
  [cc]Jul-08 16:56:53 X10Publisher  - Turning pass device (A2) on; fail device (A1) off ...
  [cc]Jul-08 16:56:56 Project       - Project dms:  idle
  [cc]Jul-08 16:56:56 Project       - Project dms:  next build in 1 minutes

At that point, the green lava lamp should turn on. Bask in that pleasant glow for a moment. When the lamp gets warmed up, you’ll get entertained by happy, green bubbles:

And then at some point somebody might check in code that causes the build to fail. (Hey, it happens to even the best programmers once in a while.) Here’s the early warning sign that the scheduled build is in trouble:

Eek! Notice that the ambient light in the red lamp goes on immediately. It will take a while for the red lava to heat up and start to boil. In the meantime, there’s work to be done. When you check the CruiseControl log, you’ll see something like this:

  BUILD FAILED
  file:C:/work/dms/builds/checkout/dms/build.xml:77: Tests failed! Check test reports.

  Total time: 5 seconds
  [cc]Jul-08 16:58:20 Project       - Project dms:  merging accumulated log files
  [cc]Jul-08 16:58:20 Project       - Project dms:  publishing build results
  [cc]Jul-08 16:58:20 X10Publisher  - Turning pass device (A2) off; fail device (A1) on...
  [cc]Jul-08 16:58:23 Project       - Project dms:  idle
  [cc]Jul-08 16:58:23 Project       - Project dms:  next build in 1 minutes

If it takes too long to fix the build, the red lamp will start to get angry:

The object of the game is to keep the green lamp glowing. :-)

Hearing the Build Break

You may have noticed that when the transceiver module turns on and off it makes a fairly loud snapping sound. That sound is caused by the mechanical relay inside. (The lamp module doesn’t make that sound, I suspect because it’s a low-current device that doesn’t use a mechanical relay.)

This audible feedback turns out to be quite handy if your team is working in close quarters. The "snap" could be yet another sound in your project soundscape. By plugging the red lamp into the transceiver module (A1), you’ll hear that snapping sound just before the red lamp turns on. So if you’re heads down (without headphones), the sound will alert you to a build failure. Indeed, you’ll hear the build break!

Important Safety Notes

These lamps get insanely hot. If you don’t let them cool down for at least five minutes before touching the glass, you’ll find out just how hot. (Yes, even the green one is hot.)

The instructions for my lamps note that they perform best if operated for less than ten hours at a time. Your local fire marshal would certainly agree. But who will remember to turn off the lamps at the end of every programming day? Well, that’s a job for the ol’ computer. Just write a shutdown program that uses the Pragmatic Automation X10 software to turn off the lamps. For example:

  new X10Device("COM1", "A1").off();
  new X10Device("COM1", "A2").off();

Then schedule an at or cron job on the scheduled build machine that runs the shutdown program at some hour of every evening, and another program that turns the lamps on every morning.

What About My Macintosh?

The Firecracker Computer Interface plugs into a serial port. Modern day Macintosh computers don’t have a serial port. Sadly, none of the USB-to-serial converters I tried worked with the Firecracker Computer Interface. (I’d love to hear otherwise.)

You can purchase X10 kits that plug into USB, but I chose to use the Firecracker kit because it’s relatively cheap and it’s what is described in the book. Thankfully, by writing and testing most of this software against a mock device, I was able to do the majority of the development on my PowerBook. :-)

Conclusion

It’s relatively easy and inexpensive to make build monitoring a spectator sport. Put the lava lamps in a highly-visible area of your project and use them as visual (and audible) feedback devices, and to show off your programming prowess, of course. And that’s just the beginning. You can use any appliance to monitor anything that’s valuable to your team.

Are you using feedback devices to spice up your project? If so, please share your story.

Enjoy!

Screeching Animatronic Monkeys

Suppose you’ve just launched a new web application. Being a savvy automator, you decide to set up monitors for significant system events like, say, new users signing up. Now it’s always good to pick a physical device that gives visual or audible feedback that in some way matches the logical event you’re monitoring. So what kind of feedback goes along with a new user signing up?

Ian Bishrey couldn’t exactly wire up a human, so he went with the next best thing: a screeching Wowwee Alive! Monkey.

You know you want one of these on your project, and Ian has instructions and code for how he hacked the remote controller and wrote a Java-based serial interface (with tests) to send commands to the animatronic monkeys.

In the email he sent, Ian concluded by stating matter of factly "Using the remote means we can control an army of chimps simultaneously." I had to read that sentence a couple times, and each time the mental images got progressively more frightening. Added to which, Ian mentioned in passing that they also use the monkeys to give feedback on "low NLP linguistic classification confidence score events". So to summarize, we have a web 2.0 application that’s taking on new users and in some way processing natural languages, all being monitored by a potential army of monkeys.

You know, when I challenged readers of the automation book to find creative ways to monitor their software projects, I never imagined it could come to this…

In the Lap of Monitoring Luxury

So you’re dying to hook up X10-enabled devices to monitor your build status, but the tireless build machine is under lock and key. Or maybe it’s not physically located in your building. Or maybe you want to use lava lamps in your office, but when your manager is at home he wants a successful build to fire up the ol’ massage chair. Ah… Yes, there are too many devices to choose from and not enough serial ports.

Ralph Jocham has come to the rescue by releasing CCScrape: a Java Web Start application that scrapes the CruiseControl results web page and turns on/off X10 devices hooked up to your local machine. It currently only runs on Windows machines with an available COM port, but if you have one of those it’s as easy as plugging in your devices and clicking here.

Get a load of that: screen scraping, build monitoring, and an auto-updated application! Ralph, you deserve a gold star for automation! :-)

Visualizing Your Network Status

jgaynor writes:

Inspired by an old Slashdot article and my obsessive need for shorter network outage response times, I decided this year to create a ‘Christmas lights’ frontend to our Network Management System. The results have been very encouraging. …

I recently moved from NOC operator to engineer and can no longer spend all my time staring at and refreshing network status webpages :(. As such I wanted something that could alert me to outages with little or no effort on my part. …

In a nutshell, the Thinkpad checks a remote cron-generated file every five minutes for the existence of any alarms. Not wanting to reinvent the wheel for this, I use data from our pre-existing and very capable network monitoring system to populate that file. The Thinkpad then turns on the corresponding circuit of Christmas lights based on what it found in that file - white for all clear and red for trouble.

Instructions, pictures, movies, and even a live web cam are here.

His Bash script is another good ol’ fashioned screen scraper, powered by wget. To toggle two X10 devices, he uses BottleRocket. Simple, elegant, and well worth every hand-typed character:

  #!/bin/sh

  /usr/bin/wget \
    http://censored-because-I-like-not-being-fired.edu/remotefile \
    -O /home/localfile

  CHECK=`grep 1 /home/localfile`

  if [ -n "$CHECK" ]
  then
    br A2 on
    br A1 off
  else
    br A1 on
    br A2 off
  fi

Put the script on a schedule, plug in monitoring devices of your choosing, and then consider what you’re going to do with all the time you’ll save.

Christmas Glow

Have you put up your Christmas lights? With each red and green strand, you can radiate the build status another 50 feet or so across your project. Jeremy Whitlock sent in these festive pictures of his project’s Christmas lights, controlled by the same X10 devices that power the lava lamps.

Automation Gear On Display

Stop by and visit the good folks at Softpro Books in Denver and you’ll get a first-hand look (and feel) at some of my project automation gear. You know you’ve arrived when you see the red and green lava lamps blazing through the storefront window.

The X10 devices that conveniently power the lamps are also there on the table, ready to be poked and prodded. And, because you know you want to hook up lava lamps to your scheduled build, take a free copy of the printed instruction booklet back to your manager’s office.

While you’re in there, please support Softpro and your dear author by purchasing a copy of the book that gives you soup-to-nuts recipes for automating your software project. When you pay, don’t forget to ask for a voucher to purchase the PDF version of the book for just 99 cents. (The PDF offer applies to any Pragmatic Bookshelf title you buy at a participating independent bookstore.)

Thanks, and have fun!

Lights, Camera, Action!

Lava lamps are visual feedback devices. But what if you can’t see your project’s lamps because they’re connected to a build machine that’s halfway around the world? Bruce Johnson relayed how one of his company’s clients share the warm glow of their lava lamps across a dislocated team:

Because the development team is located both in Canada and in Europe, they not only set up the lava lamps here. They also hooked up a web cam pointing at the lamps so that the European developers could be kept informed of the build status. (Picture)

The funny thing is, whether coincidentally or not, the state of the build seems to have improved since the lava lamps were added to the picture. The green lamp is boiling at the time the picture was taken. That’s not as obvious in the picture because the lights behind appear to be reflecting through the lamp.

For some reason, I’m now picturing completely virtual teams of developers using centrally located lava lamps and web cams. ;)

This might seem like just more eye candy on a project, but it’s actually an effective way to synchronize work across dislocated teams. Imagine you’re on the team in Europe. You show up for work in the morning about the time the Canadian team is watching hockey after a long day at the office. Before you start crafting more code, you want to see a green lamp bubbling in Canada. That means all the code in version control is compiling and passing its tests. It gives you confidence that you’re working on solid ground.

And, of course, at the end of the day you want to leave the lamps just as they were at the beginning of the day… green.

Boiling .NET Code

Curtis Olson particularly enjoyed the Monitoring chapter of the book and now has lava lamps glowing on his government agency project. He contributed this story:

I simply had to implement the lava lamp monitors. However, we write code in C# rather than Java, so I was left without a pre-written publisher for CruiseControl.NET that would control X10 devices. Dogged determination led me to develop my own publisher. It was surprisingly simple.

First, download Robert Chartier’s X10 library for .NET and use the pre-built Communications.dll and X10Unified.dll files. Then add my LavaLampPublisher.cs file to the Thoughtworks.CruiseControl.Core assembly. Finally, add the following line to the <publishers> section of your ccnet.config file:

  <lavalamp passdevice="1" faildevice="2" housecode="A" />

I even have the build machine running as a virtual machine under VMWare with the virtual COM port mapped to the physical COM port on the host server.

Your Router Is On Fire

The lava lamps article struck a cord with a lot of people interested in monitoring builds, but visual monitoring goes way beyond software development. Indeed, visualization is one way to cast the abstract into concrete terms. James Turner sent in this story:

: Nigh on 10 years ago, I had built an SNMP-based monitoring system in Perl that graphically displayed the status of all the routers and end-point devices in the network. I already had it sending me pages, but I felt something more forceful was needed for in-building notification. One trip to You-Do-It electronics later, I had a massive red rotating beacon light and associated X10 controller equipment. No one ever ignored a dead router again…

Light Up Your System Tray

If you just can’t convince the boss to buy those lava lamps, or your team is dislocated and can’t share a visual monitoring device, then you need another inexpensive way to visually monitor your build. Paul Julius sent in these two screenshots of his system tray and a note about how it works:

The tray icon is red when broken and green when successful. When the build breaks, a small popup dialog appears to let you know, followed by a similar popup when the build is "fixed". A normal successful build doesn’t trigger any popups.

As an aside, the systray icon works great as "instant notification" for teams that are not co-located. However, for co-located teams, they may find the x10 publisher to be a better route. It’s "shared instant notification".

Those red and green icons in Paul’s tray are toggled by a nifty Python script donated to the CruiseControl project by Ivan Moore. Ivan had found the optional CCTray utility in CruiseControl.NET to be very handy, so he wrote a similar utility for CruiseControl. The script monitors your build by gently scraping your project’s CruiseControl web page on a configurable interval looking for the signs of a broken build.

To light up your system tray, run:

  python cruiseTrayIcon.py -u http://yourCruiseBuildPage.html

You can also mix in these command-line options:

  -q (quiet = no dialog box on status change)
  -d pollingDelayInSeconds (how long it waits between polling cruise)
  -b buildFailedString (existence on cruise page means build broken)

The cruiseTrayIcon.py script is currently only available in the CruiseControl CVS repository. The upcoming 2.2 release of CruiseControl will include Ivan’s script, along with the other significant enhancements. Until then, to check out the script and its supporting files, use:

  $ set CVSROOT=:pserver:anonymous@cvs.sourceforge.net:/cvsroot/cruisecontrol
  $ cvs login
  (Logging in to anonymous@cvs.sourceforge.net)
  CVS password: (Press Enter)
  $ cvs -z3 co cruisecontrol/contrib/cruiseTray

Make sure to read the short README file. It tells you which versions of Python and wxPython you need to install. It also tells you how to build a self-contained .exe file that you can share with team members who don’t want to install Python and wxPython.

I like that this script uses good ol’ fashioned screen scraping. It’s simple, but very effective. To build one of these monitors, all you need to know about CruiseControl is the URL of the build status page and what text to look for on that page. So if you can’t or just don’t want to use Python, you could whip one of these up in your language of choice lickety-split. If you do, please take a moment to let us know about your creation, then kindly donate it to the CruiseControl project. And unless somebody beats me to it, I might just have to write something that lights up the status bar or the dock for us Mac users. Virtual lava lamps, anyone?

Now that you can monitor the status of your build just by glancing at your system tray, what other things might you be able to monitor on the cheap using a simple screen-scraper?

Oozing Confidence

Alberto Savoia writes:

A few months ago, on April 1(!) 2004 to be precise, I posted an article on eXtreme Feedback. The article was on a relatively serious subject: "How do you get your team to pay attention to the software/project status and metrics that you care about?", but one of my solutions for getting the team to pay attention was to "invent" and implement eXtreme Feedback Devices (XFDs) that would be very visible, fun, and hard to ignore.

One of these XFDs consists of a pair of lava lamps (one green and one red) remotely connected to our build and test system in such a way that a successful build (all tests pass) turns on the green lava lamp, and a failed build (or failed tests) turns on the red one.

The original Java lava lamps have been glowing red and green for the past several months in our offices, and have achieved something of a cult status (e.g. they are included in Mike Clark’s excellent book Pragmatic Project Automation, and have recently received a fair amount of buzz on Slashdot).

The interesting thing, for me, is that something that I started as something of a joke (it was April 1st after all) actually turned out to be a very useful tool in more ways than one. Sure, I could go to our CruiseControl page to see if they build is broken, or set-up email alerts, but keeping track of the lamps (which are centrally located in our development area) is easier, faster, and gives me an ongoing view into the current status and ebb-and-flow of our build and test cycles.

I got an opportunity to visit Alberto’s project a few weeks back and witness first-hand those infamous lava lamps. You really can’t miss them. When I walked in, the red lamp was bubbling. And yet the managers weren’t beating the programmers about the head and shoulders, as some might fear. Indeed, I didn’t sense any sort of panic or condescension. What I did sense was confidence.

See, the team had learned to take the red lamp as feedback. They run an extensive battery of tests on every build and the red lamp was telling them that their tests were actually testing something. An assertion somewhere had failed the last time somebody checked in code. This is a good thing. Indeed, this is what continuous integration is all about. It becomes a bad thing when the green lamp never turns on. So they were diligently (and confidently) working to repair the build as a priority over other tasks for the day because they didn’t want to work on unstable ground.

It’s important to note that Alberto’s team is a bright group of folks building impressive products. Feedback is serious business, but they’ve found a way to make it fun. They use the lava lamps to their advantage, not as gimmicks or another way for the pointy-haired manager to keep tabs on their work. And when the lamp does go red, they’re confident that it won’t stay that way for long.