Digital Digressions by Stuart Sierra

I once saw a TV show about competing groups of archeologists trying to demonstrate how the ancient Egyptians raised stone obelisks weighing hundreds of tons.

One group of archeologists built a complex apparatus involving a wooden frame and lots of rope. It looked impressive, but it didn’t work.

The other team built a sand pit, dragged the obelisk to the top, and gradually removed sand from below the obelisk until it reached its final position. Simple, unglamorous, and it worked on the first try. (See the whole series of photos.)

The leader of the wooden-frame group admitted they were mistaken in basing their design on the most complex ancient technology they could find, sailing ships. Instead, he said, “We should have asked, ‘What is the simplest way they could have done it?’”

***

From the ancient Egyptians, jump forward about four thousand years to me, sitting at my computer, writing a new testing framework for Clojure. I was excited about the new Clojure features datatypes and protocols. I based my whole framework around them. It was a beast. Lots of weird edge cases and complex interactions that were hard to reason about and even harder to debug. Datatypes and protocols may be a powerful tool, but that doesn’t always make them the right tool.

After my fourth or fifth rewrite of Lazytest, I started asking myself, “What is the simplest way I could do this?”

The answer was staring me in the face. Clojure is a functional language (mostly). What’s the simplest way to do anything? Functions!

All of a sudden, complexity started to fall away. My protocols, most of which only had a single method anyway, became ordinary functions. My typed data structures became ordinary maps. The code shrank by many lines, and it was vastly easier to understand. Even better, I discovered new possibilities in the simpler design.

Functions are a fantastic abstraction because they can be composed. In the new Lazytest, everything is a function: test cases, test suites, and contexts. The RSpec-like describe macros are still there, but they’re simpler. They do what macros are supposed to do: provide a convenient syntactic layer over functional definitions, not define a completely new language.

I haven’t completely finished the new API — fixtures, now renamed back to “contexts,” are not supported yet. But I’m much happier with this version than with the old one. I actually feel like this is something I’d be willing to release soon. So give it a spin and tell me what you think.

I’ve been thinking a lot about testing frameworks over the past six months, and I’m not the only Stuart doing that. Stuart Halloway, who spent some time on his own Clojure testing framework, Circumspec, recently wrote about his experiences refactoring some of the language tests included with Clojure.

One of Stu’s first points is that Clojure’s assert macro doesn’t provide enough context when an assertion fails. It doesn’t tell you why the assertion failed.

That’s exactly why I wrote the is macro in clojure.test. But, as both I and the other Stuart have noted, the macros in clojure.test are too tightly intertwined with the reporting framework. So I started working on Lazytest and the expect macro.

Stu made an improved assert that reports the current values of local variables. I’ve incorporated that feature into expect. Thanks, Stu!

***

The next takeaway from Stu’s post is that thinking about invariants — properties that must always hold — is a good way to design tests. Invariants are the basis of Haskell’s QuickCheck, which Meikel Brandmeyer adapted as ClojureCheck.

I really like the idea of invariants, although I think invariant properties are more likely to be found in highly abstract, mathematical domains than in, say, web applications. But Stu’s point is that there isn’t a good place to put invariants. Complex invariants can take a long time to evaluate, so you don’t want to run them in production code. But some assertions are cheap, and important, so you do want to run them in production code.

Stu asks if we need a “new thing.” Perhaps that new thing is assertion levels, like logging levels. Level 1 could be for simple, quick checks like argument types; level 2 for more involved consistency checks, and so on. The assertion level, rather than simply assertions on/off, could be specified at compile time.

***

There’s more to say Stu’s post, and about testing in general, but that will have to wait for another day.

A couple of weeks ago I wrote about typed assertions for Lazytest.

Like so many things, it seemed like a good idea at the time. Define typed objects for each kind of assertion (e.g., equality, instanceof). When a test fails, throw an exception with one such object as the payload. The type of the object describes the failure (e.g., not equal, not instance of) and its fields describe the components of the failure (e.g., X was not equal to Y).

It worked, but as Rich Hickey himself pointed out, it was redundant. All these typed objects have no methods, just different kinds of fields. Furthermore, most of the fields aren’t even that different. Every “failure” can be identified by a predicate and the arguments which caused it to return false.

I should have known something was wrong as soon as I wrote a Protocol with no methods. I was doing JavaBean-style programming, using objects to represent structured data. I can represent them more simply as maps.

So that’s what I’ve done. The strongly typed failures are gone. The complex code transformation and replacement in the expect macro are gone.

What’s left is a much shorter form of the expect macro that examines the assertion expression and, if it contains a normal function call, evaluates its arguments before calling the function. Effectively, it follows the same evaluation rules as Clojure itself, while holding on to the intermediate values. A failing test throws an exception with an attached map holding the original unevaluated form, the evaluated components, and the return value of the predicate.

One of my stated goals with Lazytest was to enforce a clean separation between test assertions and code that runs before/around the assertions.

The Spock framework for Java/Groovy calls these the stimulus and response, identified by the keywords when and then, respectively. I find this approach attractive, but one look at Spock’s documentation shows it’s thoroughly embedded in the imperative mode:

when:
stack.push(elem)

then:
!stack.empty
stack.size() == 1
stack.peek() == elem

In my last Lazytest update, I added the expect macro, which signals failure by throwing an exception. That was useful for adding typed “reason for failure” objects to the test results, but not good in that it led me back in the direction of allowing arbitrary code in test examples.

After todays’ commits, the expect macro is still there, but you don’t use it explicitly.  I’ve reverted to allowing only a single test expression inside the it macro. That expression will automatically be wrapped in expect.

If you really need to execute arbitrary code in an example, use the new do-it macro, and call expect explicitly.

Hopefully this is explained clearly enough in the new README.