Tag: Lisp

Command-Line Intransigence

In the early days of Clojure, I was skeptical of Clojure-specific build tools like Lancet, Leiningen, and Cake. Why would Clojure, a part of the Java ecosystem, need its own build tool when there were already so many Java-based tools?

At the time, I thought Maven was the last word in build tooling. Early Leiningen felt like a thin wrapper around Maven for people with an inconsolable allergy to XML. Maven was the serious build tool, with a rich declarative model for describing dependency relationships among software artifacts. That model was imperfect, but it worked well enough to power one of the largest repositories of open-source software on the planet.

But things change. Leiningen has evolved rapidly. Maven has also evolved, but more slowly, and the promised non-XML POM syntax (“polyglot Maven”) has not materialized.

Meanwhile, I learned why everyone eventually hates Maven, through the experience of crafting custom Maven builds for two large-ish projects: the Clojure language and its contributed libraries. It was a challenge to satisfy the (often conflicting) requirements of developers, continuous integration, source repositories, and the public Maven repository network. Even with the help of Maven books from Sonatype, it took months of trial and error and nearly all my “open-source” time to get everything working.

At the end of this process I discovered, to my dismay, that I was the only one who understood it. As my colleague Stuart Halloway put it, “Maven breeds heroes.” For end-users and developers, there’s a nice interface: Clojure-contrib library authors can literally click a button to make a release. But behind that button are so many steps and moving parts (Git, Hudson, Maven, Nexus, GPG, and all the Maven plugins) that even I can barely remember how it all works. I never wanted to be the XML hero.

So I have come around to Leiningen, and even incorporate it into my Clojure development workflow. It’s had some bumps, as one might expect from a fast-moving open-source project with lots of contributors, but most of the time it does what I need and doesn’t get in the way.

What puzzles me, however, is the stubbornness of developers who want to do everything via Leiningen. Some days it seems like every new tool or development utility for Clojure comes wrapped up in a Leiningen plugin so it can be invoked at the command line. I don’t get it. When you have a Clojure REPL, why would you limit yourself to the UNIX shell?

I think this habit comes partly from scripting languages, which were born at the command line, and still live there to a great extent. But it puzzled me a bit even in Ruby: if it takes 3 seconds to for rake to load your 5000-line Rails app, do you really want to use rake for critical administrative tasks like database migrations? IRB is not a REPL in the Lisp sense, but it’s a pretty good interactive shell. I’d rather work with a large Ruby app in IRB than via rake.

Start-up time remains a major concern for Leiningen, and its contributors have gone to great lengths (sometimes too far) to ameliorate it. Why not just avoid the problem altogether? Start Leiningen once and then work at the REPL. Admittedly, this takes some discipline and careful application design, but on my own projects I’ve gotten to the point where I only need to launch Leiningen once a day. Occasionally I make a mistake and get my application into such a borked state that the only remedy is restarting the JVM, but those occasions are rare.

I pretty much use Leiningen for just three things: 1) getting dependencies, 2) building JARs, and 3) launching REPLs. Once I have a REPL I can do my real work: running my application, testing, and profiling. The feedback cycles are faster and the debugging options much richer than what I can get on the command-line.

“Build plugins,” for Leiningen or Maven or any other tool, always suffer from running in a different environment from the code they are building. But isn’t one of the central tenets of Lisp that the compiler is part of your application? There isn’t really a sharp boundary between “build” code and “application” code. It’s all just code.

I used to write little “command-line interfaces” for running tests, builds, deployments, and so on. Now I’m more likely to just put those functions in a Clojure namespace and call them from the REPL. Sometimes I wonder: why not go further? Use Leiningen (or Maven, or Gradle, or whatever) just to download dependencies and bootstrap a REPL, then execute builds and releases from the REPL.

It’s About the Platform

I’ve said It’s About the Libraries, and indeed, one of the major selling points of Clojure is that it can call Java libraries directly.

But there’s more to it than that.  Libraries are just one benefit to building Clojure on top of Java, or, more accurately, on top of Java the platform.

Look around you, and you’ll see that 99% of all the software in the world runs on just three platforms:

  1. Unix/C
  2. Java Virtual Machine
  3. .NET Common Language Runtime

Where did these platforms come from?  Let’s see:

  1. AT&T
  2. Sun
  3. Microsoft

Notice something?  All three were all developed by huge corporations.

Building a new platform isn’t just about writing the code.  In fact, very little of it is about code.  You need books, articles, conferences, workshops, and university courses.  You need multinational corporations to trust their entire business to your platform.  It takes millions of dollars and tens of thousands of hours of labor to create a new platform.  Think of the massive ad campaigns Sun ran for Java.  Can you do that?  Of course not.

So when you’re designing a new language, you have to build on an existing platform.  Most of the so-called “scripting” languages grew up on Unix, so they’re written in C.  Now, Unix/C is a great platform, still going strong after 40 years.  It provides powerful tools and standardized interfaces such as files, sockets, and pipes.

The problem is that each of the “scripting” languages has developed into its own mini-platform.  Perl, Python, and Ruby each define their own set of data structures for fundamental types like strings, lists, and maps.  The only “types” that Unix recognizes are text and binary.  You can’t exchange data between two languages without serializing everything to some agreed-upon format.  And you can’t do callbacks between languages below the level of a whole process.

The other problem with languages written in C is, well, C.  Pointers are hard.  Memory management is hard.  I know from bitter experience that Ruby libraries can have segfaults or memory leaks.  That just doesn’t happen in Java.

Clojure was created to leverage capabilities of Java-the-platform — garbage collection, dynamic code generation, JIT compilation, threads, locks — some of which are difficult to use effectively in Java-the-language.  To implement Clojure in C, for example, you would first have to build your own platform with these features.  That’s effectively what most Common Lisp implementations do, and they suffer because the Common Lisp world is too small to sustain its own platform.

The brilliant thing about Java-the-platform is that it allows many languages to coexist.  I can mix code written in Java, Clojure, JRuby, Jython, etc. and it’s pretty easy, because they all implement the same fundamental interfaces like java.util.List and  java.lang.Runnable.  For example, right now I have Hadoop (Java code) calling Clojure code calling JRuby code.  It all just works.

(The .NET CLR provides similar capabilities, and there is a Clojure CLR port.)

Cutting-Edge Clojure Development with Maven

I promised, in my previous post, that I would show you how to use the latest-and-greatest versions of Clojure and clojure-contrib in your Maven projects. Here’s that post.

Formos Software maintains a Maven server with nightly builds of Clojure and contrib at http://tapestry.formos.com/maven-snapshot-repository/

Here’s a complete pom.xml file with dependencies on both Clojure and clojure-contrib:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>YOUR.GROUP.ID</groupId>
  <artifactId>YOUR-PROJECT-NAME</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>YOUR-PROJECT-NAME</name>
  <dependencies>
    <dependency>
      <groupId>org.clojure</groupId>
      <artifactId>clojure-lang</artifactId>
      <version>1.1.0-alpha-SNAPSHOT</version>
    </dependency>
    <dependency>
      <groupId>org.clojure</groupId>
      <artifactId>clojure-contrib</artifactId>
      <version>1.0-SNAPSHOT</version>
    </dependency>
  </dependencies>
  <repositories>
    <repository>
      <id>tapestry.formos.com</id>
      <name>Formos Software snapshot repository</name>
      <url>http://tapestry.formos.com/maven-snapshot-repository</url>
    </repository>
  </repositories>
</project>

Yes, that’s a pile of XML. But it’s not that complicated once you break it down.  Here’s what’s going on:

Dependencies

The <dependencies> section lists the libraries our project depends on.  We have one <dependency> for Clojure (called clojure-lang in the Formos repository) and one for clojure-contrib.  We’re depending on SNAPSHOT versions, which tells Maven to follow the most recent version on a particular branch.

The current development branch of clojure-lang is called 1.1.0-alpha-SNAPSHOT. The development branch of contrib, which has never had a formal release, is just 1.0-SNAPSHOT.

How did I find these version numbers?  I just looked at the repository in a web browser.  In the org/clojure/clojure-lang directory I found directories named for each development branch, 1.0-SNAPSHOT, 1.0.0-RC1-SNAPSHOT, and 1.1.0-alpha-SNAPSHOT.  I chose the latest one, 1.1.0-alpha-SNAPSHOT.  Then I did the same with clojure-contrib.

If you look inside a branch directory like 1.1.0-alpha-SNAPSHOT, you’ll find hundreds of files, one for each daily snapshot, named with timestamps.

Repositories

The <repositories> section tells Maven where to look for JAR files to download.  We added the Formos repository by specifying its URL.

The <id> and <name> tags inside <repository> are purely for our own reference.  Maven only cares about the URL. We could have used any id and name to describe the Formos repository; those names will be used in Maven’s console logging.

Managing Dependency Versions

The problem with tracking the latest snapshot is that sometimes there’s a release that breaks your code.  It might be a bug, or it might just be a change in behavior that makes the library incompatible with previous versions.

The Versions Maven Plugin can help to alleviate this problem by “locking” dependencies to specific releases and updating them in a controlled way.

First, we have to make the Versions plugin available to our project.  Do this by adding the following lines just before the final </project> in your pom.xml:

  <pluginRepositories>
    <pluginRepository>
      <id>Codehaus</id>
      <name>Codehaus Maven Plugin Repository</name>
      <url>http://repository.codehaus.org/org/codehaus/mojo</url>
    </pluginRepository>
  </pluginRepositories>

We’ve added a “plugin repository,” which is just a Maven repository that happens to contain Maven plugins.  (Technically, you don’t need to add this if your local Maven cache already has a copy of the Versions plugin, but putting it in pom.xml ensures that other developers coming to your project have access to all the same plugins.)

Now we can use the following mvn command:

mvn versions:lock-snapshots

This modifies your pom.xml file, setting the version string of every SNAPSHOT dependency to the current snapshot timestamp.  For example, when I run this command, I end up with the following:

  <dependencies>
    <dependency>
      <groupId>org.clojure</groupId>
      <artifactId>clojure-lang</artifactId>
      <version>1.1.0-alpha-20090904.093041-38</version>
    </dependency>
    <dependency>
      <groupId>org.clojure</groupId>
      <artifactId>clojure-contrib</artifactId>
      <version>1.0-20090904.093531-59</version>
    </dependency>
  </dependencies>

Now, whenever you build the project, you know exactly which Clojure release you’re getting.

So you work with a particular release for a while, then you want to upgrade to the latest one.  No problem!  Just run:

mvn versions:unlock-snapshots
mvn -U install

The first command modifies pom.xml, replacing all the timestamped version numbers with SNAPSHOT versions.

The -U option on the second command forces Maven to check for updated versions of all the snapshot dependencies.

Note: Both lock-snapshots and unlock-snapshots create a backup file called pom.xml.versionsBackup.  To remove this file (and accept the Version Plugin’s changes to your pom.xml) run:

mvn versions:commit

Likewise, to go back to the pom.xml file you had before the Versions Plugin messed with it, run:

mvn versions:revert

Explore

There’s a lot more to the Versions plugin, and to Maven dependency management in general.  Check the documentation for details.

Of course, the example here can be combined with the clojure-maven-plugin demonstrated in my previous post.  The syntax of the combined pom.xml file is left as an exercise for the reader.

One other thing: if you really want the absolute latest, up-to-the-second, still-hot-from-the-github version of something, there’s nothing for it but to use Git submodules.  I’ll demonstrate that in another post.

<dependencies>
<dependency>
<groupId>org.clojure</groupId>
<artifactId>clojure-lang</artifactId>
<version>1.1.0-alpha-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.clojure</groupId>
<artifactId>clojure-contrib</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
</dependencies>
<pluginRepositories>
<pluginRepository>
<id>Codehaus</id>
<name>Codehaus Maven Plugin Repository</name>
<url>http://repository.codehaus.org/org/codehaus/mojo/</url>
</pluginRepository>
</pluginRepositories>

Maven’s Not So Bad: Further Thoughts on Clojure Package Management

Update Sept. 4: How to get the latest builds of Clojure & Contrib

Maven is a touchy subject. People tend to have strong opinions about it. But like it or not, it’s the de-facto standard for dependency management in the Java world. Clojure lives in the Java world, so that means we have to live with Maven.

Here are some good things about Maven:

  • “Convention over configuration.”
  • Plugins are downloaded & installed automatically.
  • Handles dependencies of dependencies.
  • Declarative configuration, not imperative like Ant.
  • Only stores one copy of each JAR, shared by all projects.

Here are some bad things about Maven:

  • XML configuration file.
  • Verbose command line options.
  • Doesn’t track latest source code of projects. It does; see comments (Thanks, Tim!)
  • First run takes forever to download all the plugins.
  • Verbose console output.

In my estimation, the good outweigh the bad. And nothing outweighs the huge fact that Maven is already there.

So let’s develop a Clojure app using Maven.

Step 1: Install Maven.

If you don’t already have it, that is. This is pretty easy, just visit maven.apache.org and follow the instructions.

Step 2: Create a new project.

Type the following at the command line:

mvn archetype:generate

Maven will ask a series of questions:

  1. archetype: At the “Choose a number” prompt, press enter to accept the default project type, maven-archetype-quickstart.
  2. groupId: Enter a name to identify yourself in the global Maven namespace. All your Maven projects will use the same groupId. This is typically a reverse domain name in the style of Java package names. For example, I could use the groupId com.stuartsierra
  3. artifactId: Enter a name to identify this specific project in the Maven repository. For example, my-great-clojure-library
  4. version: Press enter to accept the default, 1.0-SNAPSHOT
  5. package: Press enter to accept the default, which is the same as your groupId.
  6. Confirmation: Press enter.

Now you have a directory named my-great-clojure-library containing the skeleton of a new Java project.

Step 3: Configure your pom.xml.

Go into your new project directory and edit the pom.xml file. The basic information will already be filled out.

We can remove the dependency on JUnit, so delete these lines:

    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>

Then we need to add the Clojure Maven plugin, which tells Maven how to compile Clojure source code. Before the final </project> tag, add these lines:

  <build>
    <plugins>
      <plugin>
        <groupId>com.theoryinpractise</groupId>
        <artifactId>clojure-maven-plugin</artifactId>
        <version>1.0</version>
        <executions>
          <execution>
            <id>compile-clojure</id>
            <phase>compile</phase>
            <goals>
              <goal>compile</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

We also need to add Clojure itself as a dependency of our project (the Clojure Maven plugin does not do this automatically). Inside the <dependencies> tag, add the following lines:

    <dependency>
      <groupId>org.clojure</groupId>
      <artifactId>clojure</artifactId>
      <version>1.0.0</version>
    </dependency>

That’s if you want the official Clojure 1.0.0 release. If you want a cutting-edge version, I’ll explain how in a later post.

Step 4: Delete Java sources.

Your project directory comes pre-equipped with two Java source directories at src/main/java and src/test/java. You can delete both of them, unless, of course, you’re developing a mixed Clojure-Java project.

Step 5: Add dependencies.

If your project does anything interesting, chances are it’s going to depend on some external Java libraries. You can find libraries in the public Maven repositories at mvnrepository.com. Search for a library name, and it will show you the code to put in your pom.xml file.

For example, say we want to use the Apache Commons IO library. At mvnrepository.com, we find the dependency code for this library:

  <dependency>
    <groupId>commons-io</groupId>
    <artifactId>commons-io</artifactId>
    <version>1.4</version>
  </dependency>

And we can add that inside the <dependencies> section of pom.xml.

Step 6: Start coding!

Create the directory src/main/clojure. This is where all your .clj source files will go.

Follow the standard Clojure/Java convention for file names. That is, if you have a Clojure namespace called my.great.library, it should be in a file named src/main/clojure/my/great/library.clj

Step 7: Compile and install.

Run the following command:

mvn install

That will compile all your .clj source files into Java .class files, package them into a JAR, and install that JAR in your local Maven cache. On Unix-like systems, the cache should be at ~/.m2/repository/

Step 8: Live and learn.

There’s a whole lot more to learn about Maven. It’s a very flexible tool, and it can do almost anything. Yes, you will have to write some XML, but it’s really not that much.

Things I hope to cover in future posts:

  1. Using git submodules to track development versions of Clojure libraries.
  2. Running tests written in Clojure.
  3. Including .clj source files in your JAR.
  4. Creating a stand-alone JAR including all dependencies.
  5. Setting up a private Maven repository.

I hope this was a reasonable introduction to developing with Maven and Clojure, and that I have shown that Maven isn’t nearly as scary as people make it out to be. I think Maven suffered for a long time from poor documentation, but that’s changing rapidly. I found the (free) book Maven: The Definitive Guide extremely helpful.

Appendix: Complete pom.xml

Here’s the complete pom.xml file for the project I developed in this post:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.stuartsierra</groupId>
  <artifactId>my-great-clojure-library</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>my-great-clojure-library</name>
  <url>http://maven.apache.org</url>
  <dependencies>
    <dependency>
      <groupId>org.clojure</groupId>
      <artifactId>clojure</artifactId>
      <version>1.0.0</version>
    </dependency>
    <dependency>
      <groupId>commons-io</groupId>
      <artifactId>commons-io</artifactId>
      <version>1.4</version>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <groupId>com.theoryinpractise</groupId>
        <artifactId>clojure-maven-plugin</artifactId>
        <version>1.0</version>
        <executions>
          <execution>
            <id>compile-clojure</id>
            <phase>compile</phase>
            <goals>
              <goal>compile</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

Thoughts on Clojure Package Management

Update Sept. 3: Maven’s Not So Bad.

A lot of Ruby types come to Clojure and ask, “Where’s the package manager?” The answer is usually, “Maven or Ivy,” which isn’t really an answer.

I discussed this in the latter half of my Philly Lambda talk (PDF slides). The problem is that Clojure is built on Java, and any Clojure library that does something interesting is going to need some Java libraries beyond what the JDK provides.

Java has only one established dependency management system, Maven. (Ivy is an alternative, but it uses the Maven repositories.) Maven works, but it’s a big, complicated beast, built in the best giant-XML-configuration-file Java tradition. It’s also slow to accept new libraries into the public repositories. The central Maven 2 repository contains fewer than 700 libraries. Rubyforge, by contrast, lists over 8,000.

Maven seems to work well for large organizations that can benefit from setting up their own, private repositories, but it’s kind of a headache for the independent developer.

There’s a Clojure Maven plugin, some shell-based hacks like Corkscrew, and some Ivy-related code floating around, but none really provides what people want: one simple command to download and install all the dependencies for a project, without needing any XML.

What everyone wants, of course, is CPAN. Thousands of documented, tested modules for just about any task you could imagine, and quite a few you couldn’t (e.g., Acme::Buffy).

But CPAN was not created in a day. Most of its imitators (Rubygems, PEAR, Python Eggs) have failed to reach the same level of quality. Perl is also much older, and therefore more stable, than Python or Ruby. 10-year-old Perl code probably still works.

Part of this CPAN’s success, I think, has to do with the environment in it evolved. When Perl was the hot new language, running a web server was an expensive proposition. Even domain names weren’t cheap. If you were going to publish code on the web, there was a cost to doing so, either in time or money, so you wanted to make sure that it was worth publishing.

These days, when everyone has a blog and a Github account, sharing code is easy. Doing “git push” requires almost no thought, no investment of time. Why not release everything, even when it’s untested, undocumented, or unfinished?

So this weekend I started working on a package repository for Clojure. It was modeled it after CPAN, but designed to support anything that could be packaged in a JAR file, including compiled Java libraries and Clojure source code.

I got started. Then I thought, who would actually use this? Of the few dozen Clojure libraries that have been published on Github, only a handful are “production-ready.” Most aren’t even finished. Very few have been thoroughly tested. (I’m equally guilty in this regard.)

I concluded that it’s just too early. Clojure is a scarcely two years old. It just released “1.0” this year, and is still developing rapidly. The libraries are evolving equally rapidly. If you want to build a project using, say, Compojure, the best way to do it is with Git submodules.

The one place a package manager would really be useful is in downloading and installing the standard Java packages that get used in almost every project, like the Apache Commons libraries. For this, Maven/Ivy works, if not brilliantly.

Update: another Maven helper: Clojure-POM

CANCELED – Clojure Talk May 12

CANCELED: My presentation is canceled.  LispNYC will still meet at the Sunburnt Cow, 137 Avenue C, and I’ll be there to talk about Clojure.  But no slides, no video, etc.  My presentation is postponed to the June meeting.

I’ll be talking about my work with Clojure at LispNYC on the evening of Tuesday, May 12.   Time and location to be announced.   Slides and (hopefully) video available after the fact.

Possible topics:

  • Why Java + Lisp was such a great idea
  • Using Clojure with Java libraries like Hadoop & Solr
  • Deploying a web server with Clojure and Restlet
  • Clojure libraries I’ve written, such as test-is

Official announcement with time & location:
from LispNYC.org

Stuart Sierra presents: Implementing AltLaw.org in Clojure

This talk demonstrates the power of combining Clojure with large Java frameworks, such as:

  • Hadoop – distributed map/reduce processing
  • Solr – text indexing/searching
  • Restlet – REST-oriented web framework
  • Jets3t – Amazon S3

Join us from 7:00 – 9:00 at Trinity Church in the heart of the East Village. Afterward the discussion will continue at the Sunburnt Cow on 9th and C.

Directions to Trinity:

Trinity Lutheran
602 E. 9th St. & Ave B., on Tompkins Square Park
http://trinitylowereastside.org/

From N,R,Q,W (8th Street NYU Stop) and the 4,5 (Astor Street
Stop):
Walk East 4 blocks on St. Marks, cross Tompkins Square Park.

From F&V (2nd Ave Stop):
Walk E one or two blocks, turn north for 8 short blocks

From L (1st Ave Stop):
Walk E one block, turn sounth for 5 short blocks

The M9 bus line drops you off at the doorstep and the M15 is near get
off on St. Marks & 1st)

To get there by car, take the FDR (East River Drive) to Houston then go NW till you’re at 9th & B.  Week-night parking isn’t bad at all, but if you’re paranoid about your Caddy or in a hurry, there is a parking garage on 9th between 1st and 3rd Ave.

It’s About the Libraries

There’s a big ‘ol thread going on down at comp.lang.lisp about Clojure vs. Common Lisp. I’m biased, of course, but I have to say that Clojure and Rich Hickey are holding their own against some of the top c.l.l. flamers.

But all the arguments about functional programming, software transactional memory, and reader macros miss what was, for me, the biggest reason to switch to Clojure. It’s about the libraries, stupid. Building on the JVM and providing direct access to Java classes/methods was the best decision in Clojure’s design. ‘Cause if it’s ever been done, anywhere, by anyone, someone’s done it in Java. Twice.

A few years ago, I tried to solve the Common Lisp library problem by writing a bridge from CL to Perl 5, and was laughed out of town. Rich Hickey, I’m told, spent years trying to bridge CL to Java, and never got very far. But Clojure works, and it works great. It’s a Lisp with a squintillion libraries. Who else can claim that?

So, if I wanted Lisp with Java libraries, why not use Kawa or ABCL … or heck, JRuby? Those are all fine projects, but they all suffer from mismatches between the “source” language (Scheme, CL, Ruby) and the “host” language (Java). There is never a one-to-one mapping between types in the source language and types in the host language. So you end up needing conversions like jclass/jmethod/jcall (ABCL) or primitive-static-method (Kawa). (JRuby is slightly better, but only because Ruby is closer to Java than CL/Scheme.)

Clojure doesn’t have this problem because it was designed from the ground up for the JVM. Clojure strings are java.lang.String, Clojure maps are java.util.Map, even Clojure functions are java.lang.Runnable (and java.lang.Callable). This makes it supremely easy to mix-n-match Clojure code with Java libraries and vice-versa. I know, because every day I use Clojure with complex Java libraries like Hadoop, Restlet, Lucene, and Solr. Everything just works. I don’t have to write any foreign-function interfaces or bridge code. In fact, using Java libraries in Clojure is often easier than using them in Java!

Clojure may not be a programming language for the next hundred years, as Arc aspires to be. But it’s a great language if you want to get stuff done right now.

Tests Are Code

It’s interesting to see the first signs of rebellion against RSpec. I jumped on the RSpec bandwagon when it first appeared, mostly so I wouldn’t have to write “assert_equals” all the time. But while I liked and used RSpec, I don’t think it made my tests any better. If anything, they were a little bit worse. I found myself testing things that were not really relevant to the code I was writing, asserting obvious things like “a newly-created record should be empty.”

When I got interested in Clojure, one of the first things I wrote was a testing library called “test-is”. I borrowed from a lot of Common Lisp testing frameworks, especially the idea of a generic assertion macro called “is”. It looks like this:

(deftest test-my-function
  (is (= 7 (my-function 3 4)))
  (is (even? (my-function 10 2))))

This is pretty basic, but it’s sufficient for low-level unit testing. So far, I think that’s how the library has been typically used. There have been occasional requests, however, for RSpec-style syntax. I can see how this would be useful for testing at a level higher than individual functions, but I have come to believe that the added semantics of RSpec are not really necessary.

Right now, the test-is library is built on the same abstractions as Clojure itself. Tests are functions, so you can apply all the same tools that already exist for handling functions. Tests can be called by name, organized into namespaces, and composed. There is almost no extra bookkeeping code that I need to write to make all of this work.

In contrast, if I were to adopt the RSpec style, I would have to write code to call, store, and organize tests. That’s more work for me, and ultimately restricts the flexibility of the library for people who use it. Furthermore, RSpec has its own set of semantics, above and beyond the language itself, which must be learned.

This is my first experience supporting a library for anyone other than myself, and I don’t want to force anyone into a particular style. A library like RSpec is a complete environment that attempts to anticipate all possible usage scenarios, so it’s grown correspondingly complicated. I want to provide a set of small tools, that can be combined with other tools to do interesting things.

Of course, by making that decision I’m already dictating, to some extent, how the library can be used. But really, what I’m trying to do is set limits for myself. I will commit to providing a flexible, extensible set of functions and macros for writing tests. I am explicitly not trying to provide a complete testing framework. If someone wants to build an RSpec-style framework on top of test-is, more power to them. I will happily try to make test-is easier to integrate into that framework.

But there’s one other thing that struck me about that article that I linked to at the beginning — the idea of putting tests and code in the same file. I think that’s a great idea, and Clojure comes ready-made to implement it. Clojure supports the idea of “metadata” on definitions. You can attach a set of arbitrary properties to any object, without affecting the value of that object.

It’s easy to attach a test function as metadata on a definition in Clojure, but the syntax is a little ugly, and there is no easy way to remove the tests from production code. So I came up with in addition to my library, the “with-test” macro. It lets you wrap any definition in a set of tests. It looks like this:

(with-test
 (defn add-numbers [a b]
   (+ a b))
 (is (= 7 (add-numbers 3 4)))
 (is (= -4 (add-numbers -6 2))))

This is equivalent to adding metadata to the function, but the syntax is a little cleaner. I’ve also added a global variable, “*load-tests*”, which can be set to false to omit tests when loading production code.

I like having each function right next to its tests. It makes it easier to remember to write tests, and easier to see how the function is supposed to behave. So to the extent that test-is will promote a testing style, this is it. But it’s a pretty radical departure from the traditional style of testing, so I’m not sure how others will react to it.