Pragmatic Automation

Gleaning Project Feedback

You know you want to extract useful information from your project in an automated way, but which tools should you use and how much configuration will they take?

John Brugge has taken a number of popular open-source tools and knitted them together for you. He’s calling it Glean, and his goal is to help your project get quick and easy feedback by simply dropping Glean into your build system. It’s analogous to Maven plugins, but aimed at projects using Ant, and less ambitious.

Currently Glean includes scripts to drive the following tools:

CPD (copy/paste detector)
Emma (code coverage)
JavaNCSS (source code metrics)
PMD (static analysis)
StatSVN (Subversion repository activity analysis)
UMLGraph (generates UML class diagrams from code)
… and about a dozen more

The download includes basic documentation on how to set up Glean to run against a source tree, as well as sample configurations for a handful of open-source projects.

John packaged up Glean to scratch an itch for his clients, and he’s offering it back to the community as a token of appreciation for all the open-source tools. Glean is distributed under an MIT license.

Enjoy!

SwitchTower as an Automated Deployment Archetype

I’ve been remiss to have delayed mentioning SwitchTower until now. It’s an open-source utility that makes one-step deployment to one or more machines both possible and painless.

Say, for example, you need to roll out a new release of your killer app to two web servers, four application servers, and a database server. You could manually log in to each of those machines and type a list of commands, or just let SwitchTower do it for you:

  rake deploy

That single incantation causes SwitchTower to execute the commands listed in your deployment recipe in parallel (and atomically) on multiple machines. (It uses ssh to communicate with the remote machines.) It’s really that easy.

But then you notice that the application you just deployed has a bug. After giving your manager fifty push-ups, you’re thankful you can press the "undo" button:

  rake rollback

That command rolls back the application to the last deployed version on all deployed machines. Yes, it’s really that easy.

I’ve been starting to use SwitchTower on all my Rails projects, and demo’ing it in our Pragmatic Rails Studio. As a result, SwitchTower has truly spoiled me. It’s an automator’s dream, and I can’t imagine going back to manual deployment gyrations.

That brings us to the point of this elevator pitch: Can you deploy and roll back software releases this easily and consistently on your project? If not, I’d recommend you give the SwitchTower manual a read, especially if you aren’t building Rails applications. That’s right, even if you’ve never heard of Ruby or Rails you owe it to yourself to check out SwitchTower because its concepts aren’t specific to Ruby or Rails. And in fact it seems possible to use SwitchTower as the deployment vehicle for apps of any size and color by customizing a core set of tasks. I’d love to hear of efforts to adapt SwitchTower to J2EE projects, for example.

Every software project can benefit from a deployment process that takes less time and money without sacrificing quality. SwitchTower is an archetype of automated deployment. Check it out before rolling your own solution.

CruiseControl 2.3.1 Released

CruiseControl 2.3.1 was released today with an enviable display of nips and tucks.

Get it here.

CruiseControl 2.3 Released

The dedicated team of folks who write code to keep your code continuously integrated have obviously been busy this summer. CruiseControl 2.3 was released yesterday with a goodly amount of new features and plugins (release notes).

Get it here.

Tiger: An Automation Platform

In the last 24 hours, I’ve upgraded my private and public worlds. Thankfully, both upgrades went off without a hitch. This PowerBook is now happily running Tiger and my company web site is now being hosted by TextDrive. The end result is that I now have more options for automation across the board.

On the global front, TextDrive has all the tools I need to get things done on the web. It’s nothing short of a geek’s paradise, and the support has been superb.

Locally, Tiger has all the fixings for an automation feast: Automator, Dashboard, and even the latest version of Ruby. Automator and Dashboard have been primarily focused at the consumer market, but both have a lot of potential for project automation. Imagine a Dashboard widget that shows, at a glance, the health of your build or production application. As well, Automator is at your service to help simplify repetitive or time-consuming project chores. Imagine a consistent, repeatable workflow that takes your source code and, through a series of visually-assembled actions, transforms it into a packaged release.

Here are just a few resources to whet your appetite:

Working with Automator: An overview of Automator and primer for writing Automator actions (Apple Developer Connection)
Delegating the grunt work: An introduction to Automator (MacWorld)
Automator World: A clearinghouse for Automator workflows, actions, news, etc.
Automator Actions: Apple’s site of Automator actions, ready for the taking
Automator Programming Guide
Developing Dashboard Widgets: A jump-start to writing your first Dashboard widget (Apple Developer Connnection)
Discovering the Dashboard : An introduction to Dashboard (MacWorld)
Dashboard Programming Guide

I’m looking forward to building a few Automator actions and Dashboard widgets myself. Moreover, I’m excited to see the killer applications that emerge from what promises to be a groundswell of activity around these tools. So please dig in, and share what you find!

Getting Started with Rake

Jim Weirich has started writing what promises to be an excellent series of Rake tutorials (Getting Started and Handling Common Actions).

Rake is a Ruby build program that rhymes with make, but for all intents and purposes that’s where the similarity ends. Rake’s built-in tasks read like a mini build language and because Rakefiles are defined in standard Ruby syntax, you have all the power of a genuine object-oriented language with which to do automation galore.

Yum.

Subversion: It's What's For Dinner

Project automation relies on version control for consistently and repeatability. If you’re not using version control, or your version control system is getting in your way, then the answer has never been clearer: Subversion. It’s free, it works great, and it stays out of your way.

Mike Mason, master chef of tasty Subversion dishes, has put together a collection of his best recipes to help you get up and running with Subversion quickly, and correctly. His book, Pragmatic Version Control Using Subversion, just went on sale. Get it while it’s hot!

CruiseControl 2.2.1 Released

CruiseControl 2.2.1 is now available, thanks to Jeffrey Fredrick and all the other hard-working folks named in the release notes.

New features since 2.2 that caught my eye include:

A configuration GUI was added to the contrib directory for standalone use or deployed with Java Web Start (JNLP file).
CM Synergy is now supported—a notable new feature for two projects I recently visited.
Default attributes can be specified for plugins to help whittle down the size of your config.xml file.

And that’s just the tip o’ the iceberg from my vantage point. In addition, many fixes were contributed from the community.

Off the Treadmill, Onto the Rails

If you haven’t heard of Ruby on Rails, it might be time to look up from your keyboard. Rails is an open-source web framework in Ruby, and it’s gaining mindshare at a staggering pace. But people aren’t just talking about Rails; they’re doing it, using Rails to build useful web applications. Indeed, the number of Rails applications that have been released in the last several weeks—applications powered by just a couple hundred lines of code—speaks well for the productivity boost of Rails development.

I caught up with David Heinemeier Hansson, the creator and tireless evangelist of Rails, to answer a few quick questions about the automation aspects of Rails.

Mike: Everybody who tries Rails raves about how it makes them super productive. What types of automation does Rails employ to help developers create great apps so quickly?

David: The basic philosophy is to encourage good behavior through invitations. So, for example, when you use the generator to create a new model or controller, it also creates unit test stubs that are all hooked up. You just enter a new test case and off you go. The same goes to fixtures, where a YAML file is already created and hooked up, just waiting for you to input the data.

Additionally, we have the entire test process wired up through Rake (Make for Ruby). So by default, if you just enter rake in a new Rails project, it’ll run all the unit and functional tests. Oh, and it’ll give you a statistical report about your code base, too. Like code-to-test ratio and stuff like that.

Mike: Sounds like you’ve tried to automate everything that will be the same on most projects.

David: We basically try to make it so that the only thing you need to create is application-specific functionality. All the infrastructure is already configured and ready to run. That means it’s much, much easier to get off to a good start with testing, revision control (we backup your DB schema for you), and the like.

So even though we call it a Rails skeleton, when you create a new empty application, it’s actually more a fully fleshed, but undressed body you’re dealing with (yeah, Rails is that sexy :)).

Mike: I like the fact that the automation built into Rails—code generation, templates, one-step test process, etc.—invites good behavior and productivity. It makes it easy for people to do the right thing. Speaking of testing, what are you using for continuous integration with Rails and Rails projects?

David: DamageControl, which is a project out of ThoughtWorks by Aslak and Tirsen. It’s a bit fragile at the moment, but still very useful. It has already saved me a couple of times. I’m certainly a big believer in continuous integration. We have it set up for Basecamp and two recently announced Rails projects: Ta-da List and Writeboard.

Mike: Rails projects are popping up all over. I just started converting a small J2EE project to Rails, and let me just say that Rails is highly addictive stuff. You mention DamageControl, a web application that’s already written in Ruby. Any chance it will be converted to a Rails application?

David: Yes, I believe Aslak from the DamageControl team is working on this already. DamageControl was using its own controller framework-like setup, but seeing as Action Pack is readily available and will allow the project to run on Apache as well as WEBrick, it seemed like a great fit.

I’m really excited to see DamageControl come to Rails. I’m sure it’ll mean an influx of good ideas and code from the existing Rails community. We’re quite eager to see it happen.

Thanks for taking time out to share your thoughts, David. Keep up the great work on Rails!

A Sneak Peek at Automator

I enjoy writing programs that glue recurring tasks together into an automation workflow. And as a programmer, I have lots of automation tools within reach. But those tools often only get me half way to my goal. Once I’ve written a program that automates a task, I’d like to share the task with non-programmers and allow them to use it to build their own automation workflows. Indeed, automation is for all of us.

The goal of ubiquitous automation is one of the reasons I’m excited about Automator: a feature in the soon-to-be-released Mac OS X Tiger. To my mind, it enables automation for the masses. Simply by dragging preloaded Actions onto a Workflow, anyone on your project can automate the tasks of checking out code from version control, building the project, and creating an installation package. That’s a few project chores you can check off the list, but imagine what other chores you might be able to automate when you start creating and sharing your own custom Actions.

Working with Automator, the latest article in James' Tiger Developer Overview Series, gives you a quick overview of Automator, and then shows you how you can build your own Actions and share them with other users.

Automator not only puts another automation tool within your reach, but it also conveniently puts the automation tools you build within the reach of others. When you start playing with Automator, please do write in with your stories. I plan to share any Actions I craft right here, so stay tuned.

CruiseControl 2.2 Released

CruiseControl 2.2 is now available. This release adds a boatload of new features and squashes known bugs.

I continue to be impressed by all the hard work and pride in workmanship that goes into this project and the expert support provided by the CruiseControl community. Just take a look at all of the individuals listed in the release notes who contributed to this release. There are countless others who help folks on the mailing list every day. Thank you all for making CruiseControl a great continuous integration tool!

Tracking Projects by the Dashboard Lights

Tom Copeland wrote an insightful article a while back describing how his team uses automation—powered by Ruby and Java—to keep a large, geographically distributed Java project on track. He writes:

: …the UltraLog project is written in Java, by developers from over a dozen companies distributed around the United States. We needed something to help avoid integration problems; we needed a status page. So we put together the "Dashboard."

He goes on to describe how they use Ruby to parse the output of several Java-based open source tools and aggregate project metrics into a consolidated view. Nowadays, depending on the situation, I’d probably use the CruiseControl framework to publish this information on a custom tab of the build status page. But it’s neat to see how Tom’s team used a scripting language to knit all of this information together.

Programming Ruby 2 Available For Pre-Order

Dave says the book Programming Ruby 2 (aka the PickAxe book) is now available for pre-order.

Why mention a Ruby book? Well, Pragmatic Project Automation includes a Ruby script and some recent posts on this site have included Ruby code. And it turns out that Ruby is an excellent language for automating tasks, among other things. Indeed, I find myself reaching for Ruby more and more when I’m building automation tools. The language is concise and elegant, it gives you easy access to the power of your operating system, and it stays out of your way. So you’ll be seeing more Ruby code posted here.

For those reasons, and my unabashed opinion that the book is outstanding, I highly recommend pre-ordering a copy. In addition to learning a fun new language, learning Ruby has made me a better programmer in general. So even if your project has no intentions of using Ruby, you’ll be honing new skills while discovering ways to improve on old skills.

Quartz Quick Start

When I need to turn commanded automation into scheduled automation, I generally reach for cron. I’ve gotten used to its syntax and other peculiarities over the years, so cron is a tool I’m comfortable wielding. But recently I’ve been writing an automated monitoring program for a client requiring a scheduler that can live inside of their Java application and run on heterogeneous platforms. Java ports of cron are available, but I needed something with a bit more horsepower and flexibility.

Quartz is an open source job scheduling system written in Java that can be embedded within a Java/J2EE application or run as a stand-alone application. And yet those qualifications alone didn’t convince me to add the Quartz JAR file to my project. (I’m fairly picky about what libraries I’ll introduce into a project.) What did convince me to use Quartz is it makes the simple things easy and the complex things possible.

For now, I just need to do simple things: schedule recurring jobs. And before I can do the complex things, I learn by doing the simple things first. So I thought I’d share how I got started using Quartz as a way of introducing it to you. As a result, you’ll have another automation tool within reach.

Download and Install

Start by downloading Quartz.

Installation is a simple matter of adding quartz.jar and commons-logging.jar to your classpath (ok, so it’s two JAR files). You’ll find various other JAR files in the Quartz distribution, but thankfully those two JAR files are all you need to get started.

Create A Job

Next, you need to write some Java code to create a job and put it on a schedule.

Say, for example, you’re developing a build scheduler that runs Ant build files at regular intervals. (The world certainly doesn’t need another Java build scheduler with great continuous integration tools such as CruiseControl already there for the taking, but it’s an example that’s hopefully near and dear to your heart.)

Somewhere in guts of your build scheduler you create a job:

  JobDetail jobDetail =
      new JobDetail("buildJob", "buildGroup", BuildJob.class);

The JobDetail instance represents a schedulable job. The first two parameters uniquely identify the job by its name and group. The third parameter refers to a Java class that implements the Job interface. We’ll actually write the BuildJob class a bit later. But first, notice that the JobDetail refers to a class, which means you can’t call any instance methods to set the state of a Job instance. And because the scheduler will create a new instance of BuildJob each time the job is executed, the job’s instance variables are cleared between executions.

That appears to be a problem because you want your build scheduler to run a specified Ant build file and target. Therefore, when an instance of your BuildJob class is created it will need to be supplied two variables: the build file and target. Quartz has a mechanism for shuttling data to jobs when they’re run. All you need to do is put the data in the JobDataMap associated with the JobDetail instance:

  jobDetail.getJobDataMap().put("buildfile", "build.xml");
  jobDetail.getJobDataMap().put("target", "test");

The JobDataMap behaves like a Java Map with a key-value pair. We’ll see the data in the JobDataMap appear on the other side when we write the BuildJob class.

Tune A Trigger

Next, define when the job should run. Let’s say you want the Ant build file to run every 60 seconds indefinitely. To do that, create a trigger with the schedule properties:

  Date startTime = new Date();    // immediately
  Date endTime = null;            // forever
  long repeatInterval = 60000L;   // 60 seconds
  int repeatCount = SimpleTrigger.REPEAT_INDEFINITELY;

  SimpleTrigger trigger =
    new SimpleTrigger("buildTrigger", Scheduler.DEFAULT_GROUP,
                      startTime, endTime,
                      repeatCount, repeatInterval);

The SimpleTrigger is good for simple interval-based schedules. It runs the job at a specific moment in time and optionally repeats running it on an interval. Alternatively, you can use the CronTrigger to run a job on a calendar-like schedule. (Now you know another reason I like Quartz.) For example, to run the Ant build at 2:30 am, Monday through Friday, use:

  CronTrigger trigger =
    new CronTrigger("buildTrigger", Scheduler.DEFAULT_GROUP,
                    "0 30 2 ? * MON-FRI");

If you’re not familiar with cron syntax, that funny string supplied as the last parameter is a cron expression with the following form:

  Seconds Minutes Hours DayOfMonth Month DayOfWeek

Cron expressions give you a lot of scheduling flexibility. For a full range of examples, refer to the CronTrigger JavaDoc.

Schedule The Job

Until now we have not associated the job with its trigger. Indeed, jobs and triggers are decoupled. One benefit of this design is a job can be assigned to many triggers. Now we’re ready to schedule the job.

First, create a scheduler:

  SchedulerFactory factory = new StdSchedulerFactory();
  Scheduler scheduler = factory.getScheduler();

Then schedule the job by associating the job and its trigger:

  scheduler.scheduleJob(jobDetail, trigger);

The schedule is set, but jobs won’t actually fire until the scheduler has been started:

  scheduler.start();

(Jobs can also be scheduled while the scheduler is running.)

At some later time, make sure to gracefully shut down the scheduler:

  boolean waitForJobsToComplete = true;
  scheduler.shutdown(waitForJobsToComplete);

Write A Job Class

Before running the scheduler, we need to write a class that implements the Job interface. This brings us back to the BuildJob class.

You put the Java code that gets executed when the job runs in the method called execute(). Here’s an example implementation of the BuildJob class:

  public class BuildJob implements org.quartz.Job {

    public BuildJob() { }

    public void execute(JobExecutionContext context)
        throws JobExecutionException {

        JobDataMap map = context.getJobDetail().getJobDataMap();
        String buildFile = map.getString("buildfile");
        String target = map.getString("target");

        try {

            AntRunner runner = new AntRunner();
            runner.run(buildFile, target);

        } catch(Exception e) {
            throw new JobExecutionException(e);
        }
    }
  }

Notice that a JobExecutionContext instance is passed to the execute() method when the job is run. From the JobExecutionContext instance you’re able to get the JobDataMap instance containing the data defined when the job was scheduled—the Ant build file and the target.

The BuildJob simply delegates to the AntRunner class—a custom class I wrote that runs the Ant build file and target. By making Job classes as thin as possible—simply delegating the real work to a class not dependent on Quartz—the job execution code is easier to test and potentially reuse elsewhere in the system.

Notice also that the only checked exception the execute() method is allowed to throw is a JobExecutionException. Thus, any unhandled exceptions (checked or unchecked) raised during job execution should be caught and rethrown as a JobExceptionException.

Advanced Scheduling

We’ve barely tapped into the potential of Quartz. Thankfully, although it lets you do complex things, you can get started quickly and then wade into advanced features if you need them. If you want to dig deeper, the tutorial goes into more detail without overwhelming you with options.

Have fun!