Pragmatic Automation

Rake: Not Your Father's Build Language

Martin Fowler has written an excellent article describing the strengths of using Rake as a build language, and how it compares to our old friends make and Ant.

Having used Rake extensively in recent months on a Rails project, I can’t underscore enough how handy it is to have a full-blown language at your fingertips when crafting a build system. I got a taste of this power when I experimented with using Groovy to script Ant. I didn’t get a chance to push those boundaries before moving on to Rake, but it’s the same concept: build systems have much to gain from the use of an internal domain-specific language.

A must read!

Is Your Build Agile?

SD Times just posted a new article concerning lengthy builds siphoning energy off agile processes.

I think we’ll continue to see the tools and processes of old getting in the way of the rapid feedback required to move quickly and confidently. It’s an opportunity to revisit the value of those tools and processes, and adjust accordingly.

Firefox Plugin for CruiseControl

Inspired by the recent Dashboard widget, Dmitri Maximovich wrote a Firefox plugin to monitor builds in CruiseControl. He’s looking for your feedback to help make it even better…

CruiseControl Widget

The Dashboard in Tiger is an ideal way to monitor automation at work with just a glance. I’ve been anxiously awaiting word of attractive widgets that monitor builds and running programs.

Jason Sypher and Mark Husson came through with style today by unveiling the CruiseControl Widget. Check out the screenshots. Yum!

Project Dependencies Using Ant

Joe Schmetzer has posted a short and sweet article on how he uses Ant target dependencies and the <import> task to manage the build order of sub-projects. Examples scripts are there for the taking.

Ant 1.6.3 Available

Ant 1.6.3 is now available for your automation pleasure. This release includes a beefy list of fixed bugs and enhancements, some of which are related to the new subant, macrodef, and import tasks we recently toured.

Ant 1.6 Goody:

So far we’ve looked at <import> and <macrodef>. As a result, hopefully your Ant build files have lost a few pounds of angle brackets so that they’re easier to maintain. It’s a good start, but if your project consists of several sub-projects, then there’s one more new feature in Ant 1.6 that you won’t want to miss.

The <subant> task lets you uniformly call an Ant target on multiple sub-projects, without having to maintain a list of those sub-projects. This is especially helpful when you want a master build file to build all your sub-projects.

Say, for example, your project is structured as follows:

  rodents
    |
    +- chipmunk
    |    |- build.xml
    |
    +- groundhog
         |- build.xml

Each of the sub-projects, chipmunk and groundhog, can be built independently of the other. For example, to create a JAR file for the chipmunk project, you type

  $ cd chipmunk
  $ ant dist

The resulting JAR file is created in the chipmunk/dist directory.

  chipmunk
    |
    |- build.xml
    |
    +- dist
        |- chipmunk.jar

That’s all well and good when you’re working on chipmunk, but you’d also like to be able to build all the rodent sub-projects at once from a master build file. (All your rodents are belong to us.) And when those sub-projects are built, you want their resulting JAR files to be created in the top-level rodents/dist directory. That is, you want to call the jar target on each sub-project and override the dist directory so that you end up with

  rodents
    |
    +- dist
        |- chipmunk.jar
        |- groundhog.jar

To do that, create a master.xml file in the top-level rodents directory and define a jar target that uses a <subant> task.

  <project name="master">

    <property name="dist.dir" location="dist" />

    <target name="jar">
      <mkdir dir="${dist.dir}" />
      <subant target="jar">
        <property name="dist.dir" location="${dist.dir}"/>
        <fileset dir="." includes="*/build.xml"/>
      </subant>
    </target>

  </project>

The jar target uses a <mkdir> task to create a root-level directory named dist—the directory we want to contain the resulting JAR files. Then the target uses a <subant> task to run the jar target of each build.xml file found in sub-directories of the root-level directory. (Note that a <fileset> is unordered, but <subant> supports other path structures.)

When Ant is run in the sub-directories, the value of the dist.dir property will be set to rodents/dist. In other words, we override the dist.dir property so that sub-projects will create their JAR files in the root-level dist directory instead of in the project-level dist directories.

You now have a transparent master build step:

  $ ant -f master.xml jar
  Buildfile: master.xml

  jar:
    [mkdir] Created dir: /Users/mike/work/rodents/dist

  jar:
    [jar] Building jar: /Users/mike/work/rodents/dist/chipmunk.jar

  jar:
    [jar] Building jar: /Users/mike/work/rodents/dist/groundhog.jar

  BUILD SUCCESSFUL

Now imagine that tomorrow you add a new rodent sub-project. What do you have to change in order for the new project to fit into the master build system? Well, as long as the new project has a jar target that puts its output in a location defined by a dist.dir property, you don’t have to change anything. The more rodents, the merrier.

The ability to transparently iterate over sub-projects, calling a target in each of their build files, is powerful. And if all of your sub-projects define a common set of Ant targets, then you can imagine a master build file that defines the same common targets. Each target in the master build file would simply delegate to the respective target in the sub-projects by using the <subant> task. And if you’re not careful, you might end up with a lot of duplication.

Thankfully, you can avoid that mess by combining the new <macrodef> and <subant> tasks. After all, the only thing that changes from one master build file target to the next is the name of the target to be called on the sub-projects. Putting all the common Ant tasks in a macro makes your master build file more maintainable:

  <project name="master">

    <property name="dist.dir" location="dist" />

    <target name="jar">
      <iterate target="jar"/>
    </target>

    <target name="clean">
      <iterate target="clean"/>
    </target>

    <macrodef name="iterate">
      <attribute name="target" default=""/>
      <sequential>
        <mkdir dir="${dist.dir}" />
        <subant target="@{target}">
          <property name="dist.dir" location="${dist.dir}"/>
          <fileset dir="." includes="*/build.xml"/>
        </subant>
      </sequential>
    </macrodef>

  </project>

When you call the iterate macro, specifying a target attribute, it executes that target on each sub-project. The <subant> task calls all the sub-projects uniformly; the <macrodef> task keeps the machinery in one place.

I hope this mini-series has given you insight into some of the powerful new features in Ant 1.6. I’m finding that I can do more with less when I use these features. Unfortunately, I’m also finding that many projects haven’t taken advantage of these new features.

Like production code, build files that contain duplication are costly to maintain. The Ant 1.6 features we’ve explored are aimed at helping you reduce duplication in your build process so that you spend less time writing and maintaining build files. The question is: When’s the last time you refactored your build files?

Ant 1.6 Goody:

Rolling on with our tour of new features in Ant 1.6, let’s look at another new task that helps reduce duplication and promotes consistency across projects. The <macrodef> task lets you create a custom task composed of existing tasks, and specify custom parameters to those tasks on a case-by-case basis. Think of it as Ant’s way of letting you write reusable methods in your build files.

Say, for example, your build file includes the following test target that taste-tests the build:

  <target name="test" depends="compile-tests">
    <junit haltonfailure="true" fork="false">
      <classpath refid="project.classpath" />
      <formatter type="brief" usefile="false" />
      <batchtest>
        <fileset dir="${build.test.dir}"
                 includes="**/*Test.class" />
      </batchtest>
    </junit>
  </target>

The <batchtest> task creates a test suite containing the tests returned by the <fileset> element—all the tests defined in *Test.class files. The resulting suite is then run by the enclosing <junit> task. Once you get all the elements and attributes set the way you like, it works like a charm.

Tomorrow you’re writing a new JUnit test, but it’s not the sort of quick unit test that a programmer on a deadline would want to run frequently as part of the build process. No, your new test belongs in a different test suite than the quick tests. So you decide that you need a new Ant target called test-database that runs all of the database-related tests. That target will call all the same tasks as the test target, but with three differences: 1) the <fileset> will collect only the database tests 2) the value of the fork attribute will be set to true and 3) the database tests require system properties to be set.

It’s tempting to copy the test target and give it a slight makeover, but you know duplication is just a shortcut to madness. Rather, extract out the commonality into a <macrodef> task:

  <macrodef name="run-tests">
    <attribute name="classes" />
    <attribute name="fork" default="false" />
    <element name="options" optional="true" />
    <sequential>
      <junit haltonfailure="true" fork="@{fork}">
        <classpath refid="project.classpath" />
        <formatter type="brief" usefile="false" />
        <batchtest>
          <fileset dir="${build.test.dir}"
                   includes="@{classes}" />
        </batchtest>
        <options />
      </junit>
    </sequential>
  </macrodef>

Notice that the run-tests macro has two attributes: classes and fork. When you call the macro, you must provide a value for classes. The fork attribute, if not specified, has a default value of false. The real work happens in the sequential task, composed of the same tasks as your original test target. The difference here is instances of @{fork} and @{classes} are replaced with their corresponding attribute values. Additionally, the <options> element is replaced with the elements enclosed in an options element, if specified when the macro is called.

Now change the test target to simply call your run-tests macro:

  <target name="test" depends="compile-tests">
    <run-tests classes="**/*Test.class" />
  </target>

Note that these tests run with a fork attribute value of false and without any options. That is, the default attribute values are used and the optional elements aren’t provided. Ah… less XML for you to type and maintain.

Finally, define a new target that calls the run-tests macro to run just the database tests:

  <target name="test-database" depends="compile-tests">
    <run-tests classes="**/TestDB.class" fork="yes">
      <options>
        <sysproperty key="db.url" value="${db.url}" />
        <sysproperty key="db.user" value="${db.user}" />
        <sysproperty key="db.password" value="${db.password}" />
      </options>
    </run-tests>
  </target>

In this case, the fork attribute is turned on and a few system properties are provided. (Note that any element that can be nested inside of a <junit> task can be specified in the <options> element.)

Extracting the common tasks and defaults out into the run-tests macro removed duplication, which means we can change the steps of the testing process in one location: the run-tests macro. This works well for common build targets that are complex in that they include several tasks and/or use several properties, such as the <javac> task. By comparison, the new <presetdef> task is similar to <macrodef>, but it doesn’t allow custom parameters to be specified. In other words, if <macrodef> is like a Java method that takes arguments, <presetdef> is like a no-arg method.

Combined with the new <import> task we visited last time, use of the <macrodef> task leads to shorter build files and consistency across projects. And by writing macros that have intention-revealing names and use intelligent defaults, your build files start to read more like a concise build language.

There’s more to come, so stay tuned for the next new feature…

Ant 1.6 Goody:

If you’re using Ant but have yet to upgrade to Ant 1.6, then you’re missing out. Just to give you a glimpse of what you’re missing, over the next several weeks I’ll post quick examples of new features in Ant 1.6 that help reduce duplication in your build process so that you spend less time writing and maintaining build files.

To kick-start the tour, let’s begin with a new feature that promotes consistency across projects. The <import> task lets you conveniently mix in tasks from other build files. Say, for example, you have a build file called build-common.xml that includes common properties and tasks, such as a compile target.

  <project name="common">
    ...
    <target name="compile" depends="prepare">
      <javac srcdir="${src.dir}"
             destdir="${build.prod.dir}"
             classpathref="project.classpath"/>
    </target>
    ...
  </project>

Tomorrow you’re breaking ground on a new Java project, and it’s up to you to define the one-step build process. Obviously it will need to compile source files, so you need a compile target. Rather than falling prey to copy/paste reuse, create a project-specific build.xml file and import the existing build-common.xml file.

  <project name="dms" basedir=".">
    <import file="build-common.xml" />
  </project>

By importing build-common.xml, the project-specific build file has inherited all the common tasks and properties. For example, you’ve barely broken a sweat typing angle brackets and you’re already compiling code:

  $ ant compile
  Buildfile: build.xml

  prepare:
    [mkdir] Created dir: /Users/mike/work/dms/build/prod
  compile:
    [javac] Compiling 4 source file to /Users/mike/work/dms/build/prod

  BUILD SUCCESSFUL
  Total time: 3 seconds

Ah, but your project needs to do some work before and after the compilation step. It’s tempting to copy the compile target and modify it for project-specific needs, but that’s only because you haven’t tasted another spice of <import>: target overriding. Simply override the compile target in your build.xml file to add more behavior.

  <project name="dms" basedir=".">

    <!-- Override common properties here. -->

    <import file="build-common.xml" />

    <target name="compile"
            depends="pre-compile, common.compile, post-compile" />

    <target name="pre-compile">
      <echo>Compiling...</echo>
    </target>

    <target name="post-compile">
      <echo>Done.</echo>
    </target>

  </project>

Notice that the project-specific compile target uses a dependency order to ensure that the local pre-compile target is invoked before the common compile target (specified as common.compile) and the local post-compile target is called at the end.

Now you have a project-specific compile step:

  $ ant compile
  Buildfile: build.xml

  pre-compile:
     [echo] Compiling...
  prepare:
  common.compile:
  post-compile:
     [echo] Done.
  compile:

  BUILD SUCCESSFUL
  Total time: 3 seconds

Nothing was compiled because everything was up-to-date from the previous build, but notice that the common.compile target was indeed bracketed with your project-specific steps.

You’re done! So, now that your build process is in place, what’s next? Well, because setting up the build didn’t take long, you probably still remember that code you need to write. And now that you’re using the <import> task, hopefully you can steer clear of the build file until next week, when we peek at another new feature.

Need an Ant Map?

Many are the build explorers who have hacked their way through the tangled undergrowth of Ant build files in search of target dependencies. A few of these brave souls have not been heard of since disappearing into the jungle with a machete raised high above their heads. We know some of them are still slogging away deep inside the jungle only because they let out a primary scream when they discover that their labyrinth is called Ant 1.6.

No doubt, the new <import> and <macrodef> tasks in Ant 1.6 are a boon when it comes to creating a modular build system and reducing duplication across build files. But the extra level of indirection created by a mixin with an overridden abstract target, for example, can quickly throw you off the trail.

Carefully lay down your machete, brave explorer, before you’re minus one good typing finger, and pick up the Ant Script Visualizer. It lets you visualize how your Ant targets and build files are related to one another by showing all the relationships between imports, targets, macrodefs, taskdefs, etc. Using GraphViz (installed separately), Ant Script Visualizer can generate an Ant target treasure map in PNG and SVN output formats, and PDF on Mac OS X (example outputs). Launch it with Java Web Start to consistently use the freshest version.

CruiseControl Action Movie

Every time I have an opportunity to talk to a group about continuous integration, I just fire up CruiseControl and act out a project episode. Not everyone ends up using CruiseControl on their project, but at least they walk away knowing what a continuous integration tool can do for them.

To help spread the word, I made a 2-minute screen video with voiceover (8MB, QuickTime) of CruiseControl in action. The experience is lossy in the sense that I’ve sacrificed quality to keep the file size to a minimum and, well, it’s not a live performance.

Let me know what you think.

Making RSS Feeds Out of CruiseControl Log Files

A gentle reader asks where to find the XSLT stylesheet I wrote to make RSS feeds out of CruiseControl XML log files. That stylesheet is described on page 29 of the book, with a footnote mentioning that the file is available on the book’s web site. Trouble is, finding the file is a bit tricky because it’s buried in the code bundle, available here.

In the builds directory of the code bundle you’ll find a buildstatus.xsl file. If you know anything about XSLT, shield your eyes. I’m sure I committed more than one XSLT sin when I wrote the file, so please feel free to school me by making it better and contributing the new-and-improved file back.

Once you’ve placed the buildstatus.xsl file in a comfy directory on the local hard drive, add an XSLTLogPublisher to your CruiseControl config.xml file, like so:

  <publishers>

    <XSLTLogPublisher
      directory="/path/to/webserver/documents"
      outfilename="your_project_build_status.rss"
      xsltfile="/path/to/buildstatus.xsl" />

  </publishers>

After CruiseControl has completed a build cycle, you can conveniently monitor your project’s build status by pointing your favorite RSS reader to a URL similar to the following:

   http://your_server:port/your_project_build_status.rss

The generated RSS file will always contain exactly one item—a summary of the last build. That is, you won’t see a history of all builds, but you do get a link back to the CruiseControl web application if you want to do build archaeology.

Enjoy!

Mini Build Server

I realize that by owning a PowerBook I’m biased, but I can’t help but think that the new Mac mini would be an ideal dedicated build server for some small to medium-sized projects that lack available machines. It’s conveniently missing everything you don’t need in a build server: display, keyboard, and mouse. Yes, the hardware you do get is proprietary, but it runs a UNIX-based operating system that’s conveniently loaded with everything you want: a rich command-line environment with shells for commanded automation, dynamic languages such as Ruby and Python that excel at project chores, other programming languages such as Java and C/C++, secure connection protocols such as ssh, cron for scheduled automation, and the Apache web server for monitoring builds and serving up RSS feeds.

At $499 for the 1.25GHz model, I suspect the Mac mini would pay for itself the first time it detects an integration problem that would have thrashed the team for an hour. Most projects will need to pop for more memory (it ships with 256MB), but even the low-end model has enough horsepower to compile code and run tests. And if the throughput of builds becomes an issue as your project grows, you could buy a bigger box or cluster a few minis together to make an inexpensive build server.

Out of the box, plugged into your network, the Mac mini is almost ready, obviously willing, and certainly able to devote its life to continuously baking fresh builds and doing other project chores. I recognize that lots of IS departments won’t be OK with this. Please choose a machine that works best for your project. The point is you really can’t afford not to have a dedicated build server, and if cost is an issue, then you might consider starting here.

Using CruiseControl with Subversion

One of the convenient things about Subversion is that it looks and feels like CVS. If you’re comfortable with CVS, then learning Subversion is a breeze.

To demonstrate the similarities, I ported the CruiseControl configuration files included in the Scheduled Builds chapter of the book to use Subversion. As you read through that chapter, you can pretty much say "Subversion" every time you read the word "CVS", while ignoring any references to CVSROOT. Likewise, any time you see the command cvs, think svn. Oh, and Subversion does support atomic commits, so the use of a quiet period with CruiseControl is less important.

For the cc-build.xml file on page 51 of the book (page 10 of the chapter PDF), replace

  <cvs command="co dms" />

with

  <exec executable="svn">
    <arg line="co /path/to/svn-repo/dms/trunk dms" />
  </exec>

There is an Ant task for Subversion, but it’s not distributed with the current version of Ant. I used the <exec> task because it gets this small job done with less configuration steps.

For the config.xml file on page 59 of the book (page 18 of the chapter PDF), replace

  <cvs localworkingcopy="checkout/dms" />

with

  <svn localworkingcopy="checkout/dms" />

The <svn> source control element is included with recent releases of CruiseControl.

And that’s all there is to it!

MobBuilding

When Mike Blake's team used their own mobile application to serve up a fresh build of that same application, they made sure they had something to help wash the dog food down. Brilliant! He writes:

"No more think, time to drink!" shouted Fernando from his seat at a door propped up on plastic sawhorses. This was my favorite thing to hear late on a Thursday at a small start-up company south of San Francisco back at the turn of the century. We developed mobile applications at ViaFone. I worked as the build manager with several talented engineers who constantly introduced me to the latest technologies including Ant and JUnit. So it was no surprise that we had fully automated checkout, build, installation and testing of our core product.

Although the build fired off automatically early every morning, there were times when colleagues needed additional builds to use the latest functionality. Since our company mission was to create applications for mobile devices, I decided to try eating a little of our own dog food. I installed a lightweight telnet application on my Kyocera Palm OS phone, had a friend poke a hole in the firewall, and with a few simple shell scripts it was born: The Mobile Build.

Thursday drink night had grown from an informal gathering of engineers to a company wide tradition. One Thursday evening at an Irish pub in downtown Frisco, I received a call from a project manager requesting a fresh build. This was a golden opportunity to debut The Mobile Build. I pulled out my stylus, logged on to the build server, and ran a shell script. A tail of the log file scrolled gracefully by on the little monochrome screen. The phone made its way around the pub. Happy engineers watched in turn as their components built and tested successfully. The Guinness tasted sweet just then. "Technically, we’re all working right now," said our leader Fernando. Well then, I thought, cheers to automation!