Blog

Beware Choosing the Most Complex Tool for the Job

I once saw a TV show about competing groups of archeologists trying to demonstrate how the ancient Egyptians raised stone obelisks weighing hundreds of tons.

One group of archeologists built a complex apparatus involving a wooden frame and lots of rope. It looked impressive, but it didn’t work.

The other team built a sand pit, dragged the obelisk to the top, and gradually removed sand from below the obelisk until it reached its final position. Simple, unglamorous, and it worked on the first try. (See the whole series of photos.)

The leader of the wooden-frame group admitted they were mistaken in basing their design on the most complex ancient technology they could find, sailing ships. Instead, he said, “We should have asked, ‘What is the simplest way they could have done it?'”

***

From the ancient Egyptians, jump forward about four thousand years to me, sitting at my computer, writing a new testing framework for Clojure. I was excited about the new Clojure features datatypes and protocols. I based my whole framework around them. It was a beast. Lots of weird edge cases and complex interactions that were hard to reason about and even harder to debug. Datatypes and protocols may be a powerful tool, but that doesn’t always make them the right tool.

After my fourth or fifth rewrite of Lazytest, I started asking myself, “What is the simplest way I could do this?”

The answer was staring me in the face. Clojure is a functional language (mostly). What’s the simplest way to do anything? Functions!

All of a sudden, complexity started to fall away. My protocols, most of which only had a single method anyway, became ordinary functions. My typed data structures became ordinary maps. The code shrank by many lines, and it was vastly easier to understand. Even better, I discovered new possibilities in the simpler design.

Functions are a fantastic abstraction because they can be composed. In the new Lazytest, everything is a function: test cases, test suites, and contexts. The RSpec-like describe macros are still there, but they’re simpler. They do what macros are supposed to do: provide a convenient syntactic layer over functional definitions, not define a completely new language.

I haven’t completely finished the new API — fixtures, now renamed back to “contexts,” are not supported yet. But I’m much happier with this version than with the old one. I actually feel like this is something I’d be willing to release soon. So give it a spin and tell me what you think.

Assertions and Invariants

I’ve been thinking a lot about testing frameworks over the past six months, and I’m not the only Stuart doing that. Stuart Halloway, who spent some time on his own Clojure testing framework, Circumspec, recently wrote about his experiences refactoring some of the language tests included with Clojure.

One of Stu’s first points is that Clojure’s assert macro doesn’t provide enough context when an assertion fails. It doesn’t tell you why the assertion failed.

That’s exactly why I wrote the is macro in clojure.test. But, as both I and the other Stuart have noted, the macros in clojure.test are too tightly intertwined with the reporting framework. So I started working on Lazytest and the expect macro.

Stu made an improved assert that reports the current values of local variables. I’ve incorporated that feature into expect. Thanks, Stu!

***

The next takeaway from Stu’s post is that thinking about invariants — properties that must always hold — is a good way to design tests. Invariants are the basis of Haskell’s QuickCheck, which Meikel Brandmeyer adapted as ClojureCheck.

I really like the idea of invariants, although I think invariant properties are more likely to be found in highly abstract, mathematical domains than in, say, web applications. But Stu’s point is that there isn’t a good place to put invariants. Complex invariants can take a long time to evaluate, so you don’t want to run them in production code. But some assertions are cheap, and important, so you do want to run them in production code.

Stu asks if we need a “new thing.” Perhaps that new thing is assertion levels, like logging levels. Level 1 could be for simple, quick checks like argument types; level 2 for more involved consistency checks, and so on. The assertion level, rather than simply assertions on/off, could be specified at compile time.

***

There’s more to say Stu’s post, and about testing in general, but that will have to wait for another day.

Slightly Less Typed Assertions

A couple of weeks ago I wrote about typed assertions for Lazytest.

Like so many things, it seemed like a good idea at the time. Define typed objects for each kind of assertion (e.g., equality, instanceof). When a test fails, throw an exception with one such object as the payload. The type of the object describes the failure (e.g., not equal, not instance of) and its fields describe the components of the failure (e.g., X was not equal to Y).

It worked, but as Rich Hickey himself pointed out, it was redundant. All these typed objects have no methods, just different kinds of fields. Furthermore, most of the fields aren’t even that different. Every “failure” can be identified by a predicate and the arguments which caused it to return false.

I should have known something was wrong as soon as I wrote a Protocol with no methods. I was doing JavaBean-style programming, using objects to represent structured data. I can represent them more simply as maps.

So that’s what I’ve done. The strongly typed failures are gone. The complex code transformation and replacement in the expect macro are gone.

What’s left is a much shorter form of the expect macro that examines the assertion expression and, if it contains a normal function call, evaluates its arguments before calling the function. Effectively, it follows the same evaluation rules as Clojure itself, while holding on to the intermediate values. A failing test throws an exception with an attached map holding the original unevaluated form, the evaluated components, and the return value of the predicate.

Lazytest Churn

One of my stated goals with Lazytest was to enforce a clean separation between test assertions and code that runs before/around the assertions.

The Spock framework for Java/Groovy calls these the stimulus and response, identified by the keywords when and then, respectively. I find this approach attractive, but one look at Spock’s documentation shows it’s thoroughly embedded in the imperative mode:

when:
stack.push(elem)

then:
!stack.empty
stack.size() == 1
stack.peek() == elem

In my last Lazytest update, I added the expect macro, which signals failure by throwing an exception. That was useful for adding typed “reason for failure” objects to the test results, but not good in that it led me back in the direction of allowing arbitrary code in test examples.

After todays’ commits, the expect macro is still there, but you don’t use it explicitly. I’ve reverted to allowing only a single test expression inside the it macro. That expression will automatically be wrapped in expect.

If you really need to execute arbitrary code in an example, use the new do-it macro, and call expect explicitly.

Hopefully this is explained clearly enough in the new README.

Two Steps Forward, One Step Back

Following my last post I integrated typed assertions into the master branch of Lazytest.

This makes some changes to the API. Test examples in the it macro can no longer simply return true or false. Instead, they must call the expect macro.

The Context type is gone, replaced by the Fixture protocol. More documentation to come. Basically, a Fixture is a pair of methods named setup and teardown. Fixtures are responsible for handling their own state.

The file watcher still works, but the new console reporter kinda sucks. I’ve got all this great failure information from the typed assertions, now I just need to figure out how to format it.

Typed Assertions Tell You What Hurts

One thing clojure.test did reasonably well was tell you why an assertion failed. Currently, Lazytest fails in this regard.

The problem with requiring test functions to return true/false to indicate pass/fail is that they can’t attach any additional information to a failure to explain why it failed.

I realized that function return values are insufficient for describing failure conditions. Fortunately, we’ve long had another means for functions to signal failure: typed exceptions.

Typed exceptions seem to be out of favor at the moment. Clojure itself only uses a handful of generic exception types, and defines none of its own.

It’s slightly awkward to define new exceptions in Clojure because the JVM requires any thrown exception to be derived from the concrete base class java.lang.Throwable.

Sure, you could use gen-class, but generating a stub class that maps to a Clojure namespace seems like overkill for such a simple task. All I need is something that I can throw with an arbitrary payload attached.

So I did that thing that will make everyone cringe: I wrote it in Java. All nine lines of it:

package lazytest;

public class ExpectationFailed extends Error {
    public final Object reason;

    public ExpectationFailed(Object reason) {
	this.reason = reason;
    }
}

Now I can define any number of typed objects representing different failure conditions, and I still only have to worry about catching one exception type.

Next I can write functions that test for different conditions and throw ExpectationFailed when they are not met, attaching the appropriate failure object. I can even write a macro, expect, that transforms an ordinary predicate expression into an “expectation expression” by reflecting on the code.

The expect macro fills the same role in Lazytest as the is macro in clojure.test.

Now I just need to figure out how to merge all this back in to the master branch.

A Journey of a Thousand Lines Begins with a Single Test

I have a curious obsession with testing frameworks. The first thing I do with any new programming language is try to write a test framework in it. It’s a useful exercise for exploring the metaprogramming facilities provided by any language. So in C, I use preprocessor macros; in Java, annotations; and in a Lisp, macros.

When I started playing with Clojure, there was no testing framework. So I wrote one, borrowing ideas from Common Lisp test frameworks such as LIFT and Chapter 9 of Peter Seibel’s Practical Common Lisp.

This was clojure.contrib.test-is. By virtual of being first out of the gate, it became the de facto standard testing framework for Clojure, and release 1.1 gave it an official position as clojure.test.

After seeing clojure.test used in the wild, and using it on my own projects, I found some problems. I set out to fix them in a totally new framework called Lazytest, which I have been working on since February. Lazytest has gone through three major revisions already, and will probably get at least one more before I release it.

Lazytest started with the simple desire to fix all the problems I found with clojure.test, but it evolved into an attempt to make the perfect behavior-driven development framework for Clojure, incorporating all the best ideas from TDD/BDD frameworks in other languages.

This post is an attempt to document where Lazytest is now, the thought processes that got it there, and where it’s headed.

First, I’ll cover what clojure.test did wrong (and right).

Things clojure.test got wrong:

Test code is tightly coupled to reporting. Every assertion is responsible for calling clojure.test/report, which immediately prints the result and updates a global counter for tests passed/failed. The only way to change the report output format is to rebind report while tests are running. This makes it awkward to implement alternative result formats such as TAP and JUnit XML.

Tests can only be grouped by dynamic scope. Following the style of Seibel, the only way clojure.test can combine tests into groups (other than namespaces) is to call one test within the body of another. This conflicts with the default run-tests behavior of running all tests defined in a namespace, leading to the poorly-understood test-ns-hook hack. There is no way to group tests by lexical scope.

Fixtures can only be assigned per-namespace. Fixtures were a late addition to clojure.test and were not integrated well with the rest of the design. The fact that they are globally applied to an entire namespace makes them useless for all but the simplest cases.

Fixtures rely on dynamic scope. The only way to pass values from a fixture to a test function is with dynamic binding. Not only is this awkward to use (every value shared between fixtures and tests needs a global Var) it makes test functions dependent on the dynamic context provided by the framework. Individual tests cannot be run outside of run-tests.

Code templates. This was a clever idea that didn’t pan out. clojure.template/do-template is a really complicated way to do map and never should have been promoted from clojure-contrib to Clojure proper.

Tree-walking. clojure.walk was another clever idea that didn’t pan out. It is still useful in a handful of situations, such as recursively changing all keywords to strings, but it could probably be replaced with something simpler.

Things clojure.test got right:

An explicit assertion form, a.k.a. the is macro. In most drafts of Lazytest I omitted this form, instead treating the last expression of any test body as an assertion. I wanted to discourage the use of multiple assertions in a single test, but such usage is frequently necessary when testing real-world code.

Recognizing assertions by syntactic form. The is macro uses a multimethod to dispatch on the first symbol in the assertion expression. The multimethod can generate different code for different kinds of assertions, such as equality, instance? checks, or exceptions thrown.

I had hoped that people would extend the is macro with their own assertion forms, but almost no one did. It was too hard to understand and, like the rest of clojure.test, too tightly coupled to the reporting subsystem.

Goals for Lazytest

Separation of concerns. There should be well-defined interfaces for creating tests, running tests, and reporting test results. It should be trivial to replace any of those components with another that respects the same interface.

Separation of syntax from internal representations. There should be a simple, functional interface for defining tests without any need for macros. Different text syntaxes, implemented as macros, can be layered on top of this interface.

Support for continuous testing. It should be possible to start a “watcher” process to monitor directories and re-run tests when files change.

Lexical grouping. It should be possible to combine tests into groups, with unlimited nesting, using lexical scopes.

Composable per-test fixtures. Fixtures (called “contexts” at the moment) may be attached to individual tests or groups of tests, and may be composed.

Support for tagging. Tests may be tagged with arbitrary metadata, including “skip”, “pending”, and “focus” to control which tests are run.

Useful reporting. A test failure report should include enough information to diagnose the problem without referring back to the test code. This is probably the hardest goal, partly because it is dependent on having clearly-written tests.

If I can do all this, it will be TDD-nirvana, but that’s a big if. Even though I have code for most of the pieces, making them all work together will be a significant challenge.

The basics are already there, on my Lazytest github page. Please try it out and send me any feedback you have, but be aware that everything in the code, including the test syntax, is still alpha and subject to change.

I will make a proper release at some point, but not until I am satisfied that I have implemented the proper abstractions.

Spread Thin

With the profusion of “community” web sites around today, it’s getting hard to keep track of where your “community” is.

For example, the “Clojure community” exists in 7 places:

  1. clojure.org (main documentation)
  2. Github (source code)
  3. Assembla (bug tracking)
  4. Wikibooks (more documentation)
  5. Google Groups (discussion)
  6. Google Code (downloads)
  7. Freenode (chat)

That doesn’t include the defunct Sourceforge page or the Hudson build server at build.clojure.org. Nor does it take into account the groups dedicated to specific Clojure libraries.

If I want to know what’s going on in Clojure, I have to monitor at least half a dozen channels. If I want to reach a broad spectrum of Clojure users, I have to post the same content in multiple places.

Weren’t all-in-one services like Launchpad supposed to save us from this insanity? Maybe. But they never can, because everyone has different tastes when it comes to bug tracking, version control, and mailing lists.

I wish that “community” sites, rather than trying to provide all-in-one services, would make it easier to connect with other sites. Assembla, for example, connects to Github. But relating an Assembla ticket to a Groups discussion to an IRC chat to a wiki page is all but impossible.

Better yet, I wish sites like Github offered subdomain hosting, e.g. github.clojure.org.

Keyword Arguments in Clojure, the Right Way

Update Feb. 10, 2010: I was wrong. Recent discussions indicate that placing optional arguments in-line, as in my first example, is preferred. In the future, Clojure may have destructuring support for this style. For now, this post remains a useful guide to map destructuring.

Many languages, such as Python and Ruby, allow functions arguments to be passed as name-value pairs.

People often ask for the same thing in Clojure, and they end up writing something like this:

(defn foo [a b & options]
  (let [opts (apply hash-map options)]
    ...))

(foo 1 2 :optionA 3 :optionB 4)

That works, but it’s not very efficient. Every time you call foo, it has to construct a hash-map from its arguments. That’s a fair amount of overhead for what is usually a small map of options.

A better way is to use Clojure’s map binding forms. The syntax is tricky, but very powerful.

Let’s look at the general syntax of Clojure’s local binding form, let:

(let [bindings...] expressions...)

The bindings are pairs consisting of a binding form and an initialization expression. The simplest binding pair is just name-value:

(let [a 1, b 2]
  (list a b))
;;=> (1 2)

So far, so good. You probably know that you can use a vector as the binding form to bind sequential things like lists, vectors, and even strings:

(let [[a b c] "foo"]
  (list a b c))
;;=> (\f \o \o)

Moving on: A map binding uses a map as the binding form. Well, duh. Let’s call it the binding map. (catchy, no?)

The keys of the binding map are the local variables you want to create. The values of the binding map are keys in the initialization expression. The locals will be bound to the values of corresponding keys.

How about an example?

      ; binding form      ; init expression
(let [{a :keyA, b :keyB}  {:keyA 1, :keyB 2}]
  (list a b))
;;=> (1 2)

You can use the same technique for function arguments:

(defn foo [{a :keyA, b :keyB}]
  (list a b))

(foo {:keyA 1 :keyB 2})
;;=> (1 2)

Notice that we’re calling foo with a single argument, a map. This may be slightly less pretty than having key/value pairs directly in the function arguments, but it is more efficient, since the compiler knows exactly how big the argument map is. It’s also more flexible, because we can construct the argument map elsewhere and then pass it to the function:

(def options {:keyA 3, :keyB 4})

(foo options)
;;=> (3 4)

You can also mix normal, positional arguments with map arguments:

(defn foo [a b {c :keyC, d :keyD}]
  (list a b c d))

(foo 1 2 {:keyC 3, :keyD 4})
;;=> (1 2 3 4)

In the most common case, you probably want your local variables to have the same names as the keywords in the initialization expression. Clojure has a shortcut for that, a special key in the binding map called, appropriately enough, :keys:

(defn foo [{:keys [a b]}]
  (list a b))

(foo {:a 5, :b 6})
;;=> (5 6)

You can also supply a map of default values using another special key in the binding map, the :or key:

(defn foo [{:keys [a b c], :or {c 42}}]
  (list a b c))

(foo {:a 7, :b 8, :c 9})
;;=> (7 8 9)

(foo {:a 20, :b 30})
;;=> (20 30 42)

Finally, what if you want to enforce a requirement that certain keys be present? Clojure 1.1 introduced a great way to do that with pre- and postconditions. You can set a pre-condition that will throw an exception if a particular binding is nil:

(defn foo [{:keys [a b c]}]
  {:pre [(not (nil? c))]}
  (list a b c))

(foo {:a 1, :c 3})
;;=> (1 nil 3)

(foo {:b 2})
;;=> java.lang.AssertionError: Assert failed: (not (nil? c))

There are even more features of map binding that I haven’t covered here. But this should give you enough background to understand the full documentation.