A Journey of a Thousand Lines Begins with a Single Test

I have a curious obsession with testing frameworks. The first thing I do with any new programming language is try to write a test framework in it. It’s a useful exercise for exploring the metaprogramming facilities provided by any language. So in C, I use preprocessor macros; in Java, annotations; and in a Lisp, macros.

When I started playing with Clojure, there was no testing framework. So I wrote one, borrowing ideas from Common Lisp test frameworks such as LIFT and Chapter 9 of Peter Seibel’s Practical Common Lisp.

This was clojure.contrib.test-is. By virtual of being first out of the gate, it became the de facto standard testing framework for Clojure, and release 1.1 gave it an official position as clojure.test.

After seeing clojure.test used in the wild, and using it on my own projects, I found some problems. I set out to fix them in a totally new framework called Lazytest, which I have been working on since February. Lazytest has gone through three major revisions already, and will probably get at least one more before I release it.

Lazytest started with the simple desire to fix all the problems I found with clojure.test, but it evolved into an attempt to make the perfect behavior-driven development framework for Clojure, incorporating all the best ideas from TDD/BDD frameworks in other languages.

This post is an attempt to document where Lazytest is now, the thought processes that got it there, and where it’s headed.

First, I’ll cover what clojure.test did wrong (and right).

Things clojure.test got wrong:

Test code is tightly coupled to reporting. Every assertion is responsible for calling clojure.test/report, which immediately prints the result and updates a global counter for tests passed/failed. The only way to change the report output format is to rebind report while tests are running. This makes it awkward to implement alternative result formats such as TAP and JUnit XML.

Tests can only be grouped by dynamic scope. Following the style of Seibel, the only way clojure.test can combine tests into groups (other than namespaces) is to call one test within the body of another. This conflicts with the default run-tests behavior of running all tests defined in a namespace, leading to the poorly-understood test-ns-hook hack. There is no way to group tests by lexical scope.

Fixtures can only be assigned per-namespace. Fixtures were a late addition to clojure.test and were not integrated well with the rest of the design. The fact that they are globally applied to an entire namespace makes them useless for all but the simplest cases.

Fixtures rely on dynamic scope. The only way to pass values from a fixture to a test function is with dynamic binding. Not only is this awkward to use (every value shared between fixtures and tests needs a global Var) it makes test functions dependent on the dynamic context provided by the framework. Individual tests cannot be run outside of run-tests.

Code templates. This was a clever idea that didn’t pan out. clojure.template/do-template is a really complicated way to do map and never should have been promoted from clojure-contrib to Clojure proper.

Tree-walking. clojure.walk was another clever idea that didn’t pan out. It is still useful in a handful of situations, such as recursively changing all keywords to strings, but it could probably be replaced with something simpler.

Things clojure.test got right:

An explicit assertion form, a.k.a. the is macro. In most drafts of Lazytest I omitted this form, instead treating the last expression of any test body as an assertion. I wanted to discourage the use of multiple assertions in a single test, but such usage is frequently necessary when testing real-world code.

Recognizing assertions by syntactic form. The is macro uses a multimethod to dispatch on the first symbol in the assertion expression. The multimethod can generate different code for different kinds of assertions, such as equality, instance? checks, or exceptions thrown.

I had hoped that people would extend the is macro with their own assertion forms, but almost no one did. It was too hard to understand and, like the rest of clojure.test, too tightly coupled to the reporting subsystem.

Goals for Lazytest

Separation of concerns. There should be well-defined interfaces for creating tests, running tests, and reporting test results. It should be trivial to replace any of those components with another that respects the same interface.

Separation of syntax from internal representations. There should be a simple, functional interface for defining tests without any need for macros. Different text syntaxes, implemented as macros, can be layered on top of this interface.

Support for continuous testing. It should be possible to start a “watcher” process to monitor directories and re-run tests when files change.

Lexical grouping. It should be possible to combine tests into groups, with unlimited nesting, using lexical scopes.

Composable per-test fixtures. Fixtures (called “contexts” at the moment) may be attached to individual tests or groups of tests, and may be composed.

Support for tagging. Tests may be tagged with arbitrary metadata, including “skip”, “pending”, and “focus” to control which tests are run.

Useful reporting. A test failure report should include enough information to diagnose the problem without referring back to the test code. This is probably the hardest goal, partly because it is dependent on having clearly-written tests.

If I can do all this, it will be TDD-nirvana, but that’s a big if. Even though I have code for most of the pieces, making them all work together will be a significant challenge.

The basics are already there, on my Lazytest github page. Please try it out and send me any feedback you have, but be aware that everything in the code, including the test syntax, is still alpha and subject to change.

I will make a proper release at some point, but not until I am satisfied that I have implemented the proper abstractions.