I’ve been thinking a lot about testing frameworks over the past six months, and I’m not the only Stuart doing that. Stuart Halloway, who spent some time on his own Clojure testing framework, Circumspec, recently wrote about his experiences refactoring some of the language tests included with Clojure.
One of Stu’s first points is that Clojure’s assert macro doesn’t provide enough context when an assertion fails. It doesn’t tell you why the assertion failed.
That’s exactly why I wrote the is macro in clojure.test. But, as both I and the other Stuart have noted, the macros in clojure.test are too tightly intertwined with the reporting framework. So I started working on Lazytest and the expect macro.
Stu made an improved assert that reports the current values of local variables. I’ve incorporated that feature into expect. Thanks, Stu!
***
The next takeaway from Stu’s post is that thinking about invariants — properties that must always hold — is a good way to design tests. Invariants are the basis of Haskell’s QuickCheck, which Meikel Brandmeyer adapted as ClojureCheck.
I really like the idea of invariants, although I think invariant properties are more likely to be found in highly abstract, mathematical domains than in, say, web applications. But Stu’s point is that there isn’t a good place to put invariants. Complex invariants can take a long time to evaluate, so you don’t want to run them in production code. But some assertions are cheap, and important, so you do want to run them in production code.
Stu asks if we need a “new thing.” Perhaps that new thing is assertion levels, like logging levels. Level 1 could be for simple, quick checks like argument types; level 2 for more involved consistency checks, and so on. The assertion level, rather than simply assertions on/off, could be specified at compile time.
***
There’s more to say Stu’s post, and about testing in general, but that will have to wait for another day.