Slightly Less Typed Assertions

A couple of weeks ago I wrote about typed assertions for Lazytest.

Like so many things, it seemed like a good idea at the time. Define typed objects for each kind of assertion (e.g., equality, instanceof). When a test fails, throw an exception with one such object as the payload. The type of the object describes the failure (e.g., not equal, not instance of) and its fields describe the components of the failure (e.g., X was not equal to Y).

It worked, but as Rich Hickey himself pointed out, it was redundant. All these typed objects have no methods, just different kinds of fields. Furthermore, most of the fields aren’t even that different. Every “failure” can be identified by a predicate and the arguments which caused it to return false.

I should have known something was wrong as soon as I wrote a Protocol with no methods. I was doing JavaBean-style programming, using objects to represent structured data. I can represent them more simply as maps.

So that’s what I’ve done. The strongly typed failures are gone. The complex code transformation and replacement in the expect macro are gone.

What’s left is a much shorter form of the expect macro that examines the assertion expression and, if it contains a normal function call, evaluates its arguments before calling the function. Effectively, it follows the same evaluation rules as Clojure itself, while holding on to the intermediate values. A failing test throws an exception with an attached map holding the original unevaluated form, the evaluated components, and the return value of the predicate.

Posted in Programming | Tagged , | 1 Comment

Lazytest Churn

One of my stated goals with Lazytest was to enforce a clean separation between test assertions and code that runs before/around the assertions.

The Spock framework for Java/Groovy calls these the stimulus and response, identified by the keywords when and then, respectively. I find this approach attractive, but one look at Spock’s documentation shows it’s thoroughly embedded in the imperative mode:

when:
stack.push(elem)

then:
!stack.empty
stack.size() == 1
stack.peek() == elem

In my last Lazytest update, I added the expect macro, which signals failure by throwing an exception. That was useful for adding typed “reason for failure” objects to the test results, but not good in that it led me back in the direction of allowing arbitrary code in test examples.

After todays’ commits, the expect macro is still there, but you don’t use it explicitly.  I’ve reverted to allowing only a single test expression inside the it macro. That expression will automatically be wrapped in expect.

If you really need to execute arbitrary code in an example, use the new do-it macro, and call expect explicitly.

Hopefully this is explained clearly enough in the new README.

Posted in Programming | Tagged , | 5 Comments

Two Steps Forward, One Step Back

Following my last post I integrated typed assertions into the master branch of Lazytest.

This makes some changes to the API. Test examples in the it macro can no longer simply return true or false. Instead, they must call the expect macro.

The Context type is gone, replaced by the Fixture protocol. More documentation to come. Basically, a Fixture is a pair of methods named setup and teardown. Fixtures are responsible for handling their own state.

The file watcher still works, but the new console reporter kinda sucks. I’ve got all this great failure information from the typed assertions, now I just need to figure out how to format it.

Posted in Programming | Tagged , | 3 Comments

Typed Assertions Tell You What Hurts

One thing clojure.test did reasonably well was tell you why an assertion failed. Currently, Lazytest fails in this regard.

The problem with requiring test functions to return true/false to indicate pass/fail is that they can’t attach any additional information to a failure to explain why it failed.

I realized that function return values are insufficient for describing failure conditions. Fortunately, we’ve long had another means for functions to signal failure: typed exceptions.

Typed exceptions seem to be out of favor at the moment. Clojure itself only uses a handful of generic exception types, and defines none of its own.

It’s slightly awkward to define new exceptions in Clojure because the JVM requires any thrown exception to be derived from the concrete base class java.lang.Throwable.

Sure, you could use gen-class, but generating a stub class that maps to a Clojure namespace seems like overkill for such a simple task. All I need is something that I can throw with an arbitrary payload attached.

So I did that thing that will make everyone cringe: I wrote it in Java. All nine lines of it:

package lazytest;

public class ExpectationFailed extends Error {
    public final Object reason;

    public ExpectationFailed(Object reason) {
	this.reason = reason;
    }
}

Now I can define any number of typed objects representing different failure conditions, and I still only have to worry about catching one exception type.

Next I can write functions that test for different conditions and throw ExpectationFailed when they are not met, attaching the appropriate failure object. I can even write a macro, expect, that transforms an ordinary predicate expression into an “expectation expression” by reflecting on the code.

The expect macro fills the same role in Lazytest as the is macro in clojure.test.

Now I just need to figure out how to merge all this back in to the master branch.

Posted in Programming | Tagged , , | 7 Comments

A Journey of a Thousand Lines Begins with a Single Test

I have a curious obsession with testing frameworks. The first thing I do with any new programming language is try to write a test framework in it. It’s a useful exercise for exploring the metaprogramming facilities provided by any language. So in C, I use preprocessor macros; in Java, annotations; and in a Lisp, macros.

When I started playing with Clojure, there was no testing framework. So I wrote one, borrowing ideas from Common Lisp test frameworks such as LIFT and Chapter 9 of Peter Seibel’s Practical Common Lisp.

This was clojure.contrib.test-is. By virtual of being first out of the gate, it became the de facto standard testing framework for Clojure, and release 1.1 gave it an official position as clojure.test.

After seeing clojure.test used in the wild, and using it on my own projects, I found some problems. I set out to fix them in a totally new framework called Lazytest, which I have been working on since February. Lazytest has gone through three major revisions already, and will probably get at least one more before I release it.

Lazytest started with the simple desire to fix all the problems I found with clojure.test, but it evolved into an attempt to make the perfect behavior-driven development framework for Clojure, incorporating all the best ideas from TDD/BDD frameworks in other languages.

This post is an attempt to document where Lazytest is now, the thought processes that got it there, and where it’s headed.

First, I’ll cover what clojure.test did wrong (and right).

Things clojure.test got wrong:

Test code is tightly coupled to reporting. Every assertion is responsible for calling clojure.test/report, which immediately prints the result and updates a global counter for tests passed/failed. The only way to change the report output format is to rebind report while tests are running. This makes it awkward to implement alternative result formats such as TAP and JUnit XML.

Tests can only be grouped by dynamic scope. Following the style of Seibel, the only way clojure.test can combine tests into groups (other than namespaces) is to call one test within the body of another. This conflicts with the default run-tests behavior of running all tests defined in a namespace, leading to the poorly-understood test-ns-hook hack. There is no way to group tests by lexical scope.

Fixtures can only be assigned per-namespace. Fixtures were a late addition to clojure.test and were not integrated well with the rest of the design. The fact that they are globally applied to an entire namespace makes them useless for all but the simplest cases.

Fixtures rely on dynamic scope. The only way to pass values from a fixture to a test function is with dynamic binding. Not only is this awkward to use (every value shared between fixtures and tests needs a global Var) it makes test functions dependent on the dynamic context provided by the framework. Individual tests cannot be run outside of run-tests.

Code templates. This was a clever idea that didn’t pan out. clojure.template/do-template is a really complicated way to do map and never should have been promoted from clojure-contrib to Clojure proper.

Tree-walking. clojure.walk was another clever idea that didn’t pan out. It is still useful in a handful of situations, such as recursively changing all keywords to strings, but it could probably be replaced with something simpler.

Things clojure.test got right:

An explicit assertion form, a.k.a. the is macro. In most drafts of Lazytest I omitted this form, instead treating the last expression of any test body as an assertion. I wanted to discourage the use of multiple assertions in a single test, but such usage is frequently necessary when testing real-world code.

Recognizing assertions by syntactic form. The is macro uses a multimethod to dispatch on the first symbol in the assertion expression. The multimethod can generate different code for different kinds of assertions, such as equality, instance? checks, or exceptions thrown.

I had hoped that people would extend the is macro with their own assertion forms, but almost no one did. It was too hard to understand and, like the rest of clojure.test, too tightly coupled to the reporting subsystem.

Goals for Lazytest

Separation of concerns. There should be well-defined interfaces for creating tests, running tests, and reporting test results. It should be trivial to replace any of those components with another that respects the same interface.

Separation of syntax from internal representations. There should be a simple, functional interface for defining tests without any need for macros. Different text syntaxes, implemented as macros, can be layered on top of this interface.

Support for continuous testing. It should be possible to start a “watcher” process to monitor directories and re-run tests when files change.

Lexical grouping. It should be possible to combine tests into groups, with unlimited nesting, using lexical scopes.

Composable per-test fixtures. Fixtures (called “contexts” at the moment) may be attached to individual tests or groups of tests, and may be composed.

Support for tagging. Tests may be tagged with arbitrary metadata, including “skip”, “pending”, and “focus” to control which tests are run.

Useful reporting. A test failure report should include enough information to diagnose the problem without referring back to the test code. This is probably the hardest goal, partly because it is dependent on having clearly-written tests.

If I can do all this, it will be TDD-nirvana, but that’s a big if. Even though I have code for most of the pieces, making them all work together will be a significant challenge.

The basics are already there, on my Lazytest github page. Please try it out and send me any feedback you have, but be aware that everything in the code, including the test syntax, is still alpha and subject to change.

I will make a proper release at some point, but not until I am satisfied that I have implemented the proper abstractions.

Posted in Programming | Tagged , | 5 Comments

Spread Thin

With the profusion of “community” web sites around today, it’s getting hard to keep track of where your “community” is.

For example, the “Clojure community” exists in 7 places:

  1. clojure.org (main documentation)
  2. Github (source code)
  3. Assembla (bug tracking)
  4. Wikibooks (more documentation)
  5. Google Groups (discussion)
  6. Google Code (downloads)
  7. Freenode (chat)

That doesn’t include the defunct Sourceforge page or the Hudson build server at build.clojure.org. Nor does it take into account the groups dedicated to specific Clojure libraries.

If I want to know what’s going on in Clojure, I have to monitor at least half a dozen channels. If I want to reach a broad spectrum of Clojure users, I have to post the same content in multiple places.

Weren’t all-in-one services like Launchpad supposed to save us from this insanity? Maybe. But they never can, because everyone has different tastes when it comes to bug tracking, version control, and mailing lists.

I wish that “community” sites, rather than trying to provide all-in-one services, would make it easier to connect with other sites. Assembla, for example, connects to Github. But relating an Assembla ticket to a Groups discussion to an IRC chat to a wiki page is all but impossible.

Better yet, I wish sites like Github offered subdomain hosting, e.g. github.clojure.org.

Posted in Programming | Tagged | 8 Comments

Keyword Arguments in Clojure, the Right Way

Update Feb. 10, 2010: I was wrong. Recent discussions indicate that placing optional arguments in-line, as in my first example, is preferred. In the future, Clojure may have destructuring support for this style. For now, this post remains a useful guide to map destructuring.

Many languages, such as Python and Ruby, allow functions arguments to be passed as name-value pairs.

People often ask for the same thing in Clojure, and they end up writing something like this:

(defn foo [a b & options]
  (let [opts (apply hash-map options)]
    ...))

(foo 1 2 :optionA 3 :optionB 4)

That works, but it’s not very efficient. Every time you call foo, it has to construct a hash-map from its arguments. That’s a fair amount of overhead for what is usually a small map of options.

A better way is to use Clojure’s map binding forms. The syntax is tricky, but very powerful.

Let’s look at the general syntax of Clojure’s local binding form, let:

(let [bindings...] expressions...)

The bindings are pairs consisting of a binding form and an initialization expression. The simplest binding pair is just name-value:

(let [a 1, b 2]
  (list a b))
;;=> (1 2)

So far, so good. You probably know that you can use a vector as the binding form to bind sequential things like lists, vectors, and even strings:

(let [[a b c] "foo"]
  (list a b c))
;;=> (\f \o \o)

Moving on: A map binding uses a map as the binding form. Well, duh. Let’s call it the binding map. (catchy, no?)

The keys of the binding map are the local variables you want to create. The values of the binding map are keys in the initialization expression. The locals will be bound to the values of corresponding keys.

How about an example?

      ; binding form      ; init expression
(let [{a :keyA, b :keyB}  {:keyA 1, :keyB 2}]
  (list a b))
;;=> (1 2)

You can use the same technique for function arguments:

(defn foo [{a :keyA, b :keyB}]
  (list a b))

(foo {:keyA 1 :keyB 2})
;;=> (1 2)

Notice that we’re calling foo with a single argument, a map. This may be slightly less pretty than having key/value pairs directly in the function arguments, but it is more efficient, since the compiler knows exactly how big the argument map is. It’s also more flexible, because we can construct the argument map elsewhere and then pass it to the function:

(def options {:keyA 3, :keyB 4})

(foo options)
;;=> (3 4)

You can also mix normal, positional arguments with map arguments:

(defn foo [a b {c :keyC, d :keyD}]
  (list a b c d))

(foo 1 2 {:keyC 3, :keyD 4})
;;=> (1 2 3 4)

In the most common case, you probably want your local variables to have the same names as the keywords in the initialization expression. Clojure has a shortcut for that, a special key in the binding map called, appropriately enough, :keys:

(defn foo [{:keys [a b]}]
  (list a b))

(foo {:a 5, :b 6})
;;=> (5 6)

You can also supply a map of default values using another special key in the binding map, the :or key:

(defn foo [{:keys [a b c], :or {c 42}}]
  (list a b c))

(foo {:a 7, :b 8, :c 9})
;;=> (7 8 9)

(foo {:a 20, :b 30})
;;=> (20 30 42)

Finally, what if you want to enforce a requirement that certain keys be present? Clojure 1.1 introduced a great way to do that with pre- and postconditions. You can set a pre-condition that will throw an exception if a particular binding is nil:

(defn foo [{:keys [a b c]}]
  {:pre [(not (nil? c))]}
  (list a b c))

(foo {:a 1, :c 3})
;;=> (1 nil 3)

(foo {:b 2})
;;=> java.lang.AssertionError: Assert failed: (not (nil? c))

There are even more features of map binding that I haven’t covered here. But this should give you enough background to understand the full documentation.

Posted in Programming | Tagged | 5 Comments

clojure-hadoop 1.1.0 release

I’ve made a new version of my clojure-hadoop library. Downloads are on the main page.

List of changes here. This fixes some missing pieces in the first release, and adds some more Hadoop configuration options.

Posted in Uncategorized | Leave a comment

Clojure-Hadoop 1.0.0

At long last, I have made a formal release of my clojure-hadoop library. Downloads and more information here.

The 1.0.0 release is documented, but not in exhaustive detail. Other people have used this successfully, but it may not support all possible Hadoop configurations.

Watch video of my presentation at HadoopWorld NYC.

Posted in Programming | Tagged , | Leave a comment

Agents of Swing

The title of this post would make a good name for a band.

Anyway, today I’m going to talk about Swing and concurrency and Clojure.

The Swing framework is not thread-safe. That may sound strange at first, but there’s actually some sound technical reasoning behind it.

Basically, the Swing designers realized that, in order to have a multi-threaded GUI, you need locks everywhere, which typically means you have bugs everywhere.

That’s not to say a Swing application can’t be multi-threaded. Swing merely requires that the GUI be single-threaded. In Swing, all GUI-related code must run on a special thread called the event dispatch thread, which is automatically created by the Swing framework.

By design, event listeners such as ActionListener.actionPerformed always run on the event dispatch thread. To run arbitrary code on the event dispatch thread, you can use the SwingUtilities.invokeLater method.

As was pointed out in a comment on my last post, this is how all of my example apps should be started. The invokeLater method takes a Runnable argument. In Clojure, this is easy, because all Clojure functions are Runnable. So, for example, the temp-app program should be started like this:

(SwingUtilities/invokeLater temp-app)

As a consequence of running all GUI code on a single thread, any GUI code that blocks can “freeze” the entire application. Event listeners, therefore, must execute quickly and delegate long operations like I/O to other threads.

Java 6 provides the SwingWorker class to manage long-running background tasks. Unfortunately, the implementation of SwingWorker is heavily dependent on protected methods and concrete inheritance, features that are awkward to use in Clojure.

But wait, wasn’t Clojure designed for concurrency? Yes it was, and we can replace SwingWorker with Clojure’s more-powerful tools.

In this post, I will recreate the Flipper application from the Swing tutorials.

* * *

The point of Flipper is to test the “fairness” of Java’s random number generator. It uses java.util.Random to generate random booleans, simulating a series of coin flips. The number of “heads” (boolean true) should converge to half of the total number of flips.

The Java version of Flipper (source here) does the coin flips in a SwingWorker thread, which sends intermediate results to a GUI.

Our version will use Clojure Agents, a much more flexible and powerful alternative.

Lets dig into some code. First, we need a function to create a “Flipper” agent:

(defn new-flipper []
  (agent {:total 0, :heads 0,
          :running false,
          :random (java.util.Random.)}))

This sets up the initial state of the agent, with its own random number generator, two counters, and a boolean flag to designate whether or not the flipper is running.

The flipper agent will have three actions. An agent action is just a function taking one argument, which is the current state of the agent. Whatever the function returns becomes the new state of the agent.

Additionally, in the body of the action, the special Var *agent* is bound to the agent itself. The action function can send other actions to *agent*; those actions will not run until after the current action completes.

Our flipper’s first action is calculate:

(defn calculate [state]
  (if (:running state)
    (do (send *agent* calculate)
        (assoc state
          :total (inc (:total state))
          :heads (if (.nextBoolean (:random state))
                   (inc (:heads state))
                   (:heads state))))
    state))

If the flipper is currently “running,” calculate will call the random number generator and update the counters. It will also send itself to the agent again, creating a loop.

If the flipper is not “running,” calculate does nothing, but it must still return the original state.

The remaining two actions control the “running” state of the agent:

(defn start [state]
  (send *agent* calculate)
  (assoc state :running true))

(defn stop [state]
  (assoc state :running false))

These actions just change the value of :running in the agent’s state. The start action also sends the calculate action to kickstart the calculation loop.

To watch the agent in action, try this at the REPL:

(def flipper (new-flipper))
(send flipper start)

The flipper is now running in the background. At any time, you can retrieve the current state of the flipper with (deref flipper), or simply @flipper. Try it a few times and see how the counters change, as in this sample REPL session:

user> @flipper
{:total 365067, :heads 182253, :running true, ...
user> @flipper
{:total 679283, :heads 338549, :running true, ...
user> @flipper
{:total 1030204, :heads 513767, :running true, ...

To stop the calculation, type

(send flipper stop)

Subsequent calls to @flipper will all show the same value. You can restart the calculation by typing (send flipper start) again.

We need one more function to make our flipper complete. We want to compute the “unfairness” of the random number generator, or how far it is from 0.5. This is easy:

(defn error [state]
  (if (zero? (:total state)) 0.0
      (- (/ (double (:heads state))
            (:total state))
         0.5)))

The if expression is just there to avoid divide-by-zero errors in the initial case.

Note that, although I used the same argument name, state, as in the actions, the error function is not meant to be called as an action with send. Instead, we will call it on the current state of the agent like this:

(error @flipper)

* * *

The nice thing about this flipper is that, unlike the Java version, it is completely decoupled from the GUI. We can design and test it at the REPL before writing any GUI code at all.

(The Java version could have been decoupled. But because Swing requires you to create so many classes anyway, the temptation is always there to combine GUI code with “process” code.)

So let’s get this GUI going. Unlike the Java example, which used callbacks from the flipper to update the GUI, our GUI will update at a fixed rate of 10 times per second. To do this we will use a javax.swing.Timer, which fires an event every n milliseconds.

First, some imports and a helper function to create the text fields with common attributes:

(import '(javax.swing JPanel JFrame JButton JTextField
                      JLabel Timer SwingUtilities))

(defn text-field [value]
  (doto (JTextField. value 15)
    (.setEnabled false)
    (.setHorizontalAlignment JTextField/RIGHT)))

Next, we will reuse the on-action macro from an earlier post, except I’ve renamed it with-action because my editor’s automatic indentation looks better that way.

(defmacro with-action [component event & body]
  `(. ~component addActionListener
      (proxy [java.awt.event.ActionListener] []
        (actionPerformed [~event] ~@body))))

And now we’re ready for the app itself:

(defn flipper-app []
  ;; Construct components:
  (let [flipper (new-flipper)
        b-start (JButton. "Start")
        b-stop (doto (JButton. "Stop")
                 (.setEnabled false))
        total (text-field "0")
        heads (text-field "0")
        t-error (text-field "0.0")
        timer (Timer. 100 nil)]

    ;; Setup actions:
    (with-action timer e
      (let [state @flipper]
        (.setText total (str (:total state)))
        (.setText heads (str (:heads state)))
        (.setText t-error (format "%.10g" (error state)))))
    (with-action b-start e
      (send flipper start)
      (.setEnabled b-stop true)
      (.setEnabled b-start false)
      (.start timer))
    (with-action b-stop e
      (send flipper stop)
      (.setEnabled b-stop false)
      (.setEnabled b-start true)
      (.stop timer))

    ;; Create window and layout:
    (doto (JFrame. "Flipper")
      (.setContentPane
       (doto (JPanel.)
         (.add (JLabel. "Total:"))
         (.add total)
         (.add (JLabel. "Heads:"))
         (.add heads)
         (.add (JLabel. "Error:"))
         (.add t-error)
         (.add b-start)
         (.add b-stop)))
      (.pack)
      (.setVisible true))))

The GUI has three text fields showing the total number of flips, the number of “heads,” and the error rate. The “Start” button starts the flipper; the “Stop” button suspends it. You can start and stop as many times as you like without losing the results of the calculation (another improvement over the Java version, which restarts at zero each time).

The important code is in the with-action bodies. (Don’t confuse Swing ActionListeners with Agent actions.) Remember, event listeners must execute quickly, so all they do is update the visible parts of the GUI.

The “Start” button kicks off the flipper and activates the Timer. The Timer fires every 100 milliseconds, updating the text fields from the current state of the flipper. The “Stop” button suspends both the Timer and the flipper.

Run the application like this:

(SwingUtilities/invokeLater flipper-app)

We didn’t do any fancy layout because it wasn’t necessary for the example. But if you want something prettier, here’s another version using the GridBagLayout macros from my last post:

(import '(java.awt GridBagLayout Insets))

(defmacro set-grid! [constraints field value]
  `(set! (. ~constraints ~(symbol (name field)))
         ~(if (keyword? value)
            `(. java.awt.GridBagConstraints
                ~(symbol (name value)))
            value)))

(defmacro grid-bag-layout [container & body]
  (let [c (gensym "c")
        cntr (gensym "cntr")]
    `(let [~c (new java.awt.GridBagConstraints)
           ~cntr ~container]
       ~@(loop [result '() body body]
           (if (empty? body)
             (reverse result)
             (let [expr (first body)]
               (if (keyword? expr)
                 (recur (cons `(set-grid! ~c ~expr
                                          ~(second body))
                              result)
                        (next (next body)))
                 (recur (cons `(.add ~cntr ~expr ~c)
                              result)
                        (next body)))))))))

(defn flipper-app2 []
  ;; Construct components:
  (let [flipper (new-flipper)
        b-start (JButton. "Start")
        b-stop (doto (JButton. "Stop")
                 (.setEnabled false))
        total (text-field "0")
        heads (text-field "0")
        t-error (text-field "0.0")
        timer (Timer. 100 nil)]

    ;; Setup actions:
    (with-action timer e
      (let [state @flipper]
        (.setText total (str (:total state)))
        (.setText heads (str (:heads state)))
        (.setText t-error (format "%.10g" (error state)))))
    (with-action b-start e
      (send flipper start)
      (.setEnabled b-stop true)
      (.setEnabled b-start false)
      (.start timer))
    (with-action b-stop e
      (send flipper stop)
      (.setEnabled b-stop false)
      (.setEnabled b-start true)
      (.stop timer))

    ;; Create window and layout:
    (doto (JFrame. "Flipper")
      (.setContentPane
       (doto (JPanel. (GridBagLayout.))
         (grid-bag-layout
          :insets (Insets. 5 5 5 5)

          :gridx 0, :anchor :LINE_END
          :gridy 0, (JLabel. "Total:")
          :gridy 1, (JLabel. "Heads:")
          :gridy 2, (JLabel. "Error:")

          :gridx 1, :anchor :LINE_START
          :gridy 0, total
          :gridy 1, heads
          :gridy 2, t-error

          :gridx 0, :gridy 3, :gridwidth 2, :anchor :CENTER
          (doto (JPanel.)
            (.add b-start)
            (.add b-stop)))))
      (.pack)
      (.setVisible true))))

I centered the buttons below the text fields by embedding them in a nested JPanel that spans the full width of the window.

Run this example as:

(SwingUtilities/invokeLater flipper-app2)

Enjoy!

Posted in Programming | Tagged , | 9 Comments