I dropped in to hear Rich Hickey talk about Clojure at the New York Semantic Web meetup group.  Some highlights:

• Some programs, like compilers or theorem provers, are themselves functions.  They take input and produce output.  Purely functional languages like Haskell are good for these kinds of programs.  But other programs, like GUIs or automation systems, are not functions.  For example, a program that runs continously for months or years is not a function in the mathematical sense.  Clojure is mostly functional, but not purely functional.

• Most Clojure programmers go through an arc.  First they think “eww, Java” and try to hide all the Java.  Then they think “ooh, Java” and realize that Clojure is a powerful way to write Java code.  Rich frowns upon “wrapper” functions in Clojure that do nothing but wrap a Java method.  Calling the Java method directly is faster and easier to look up in JavaDoc.

• Rich recommended a paper, Out of the Tar Pit, for a discussion of functional and relational techniques to manage state.

• Clojure’s data structures are persistent.  This isn’t persistent in the stored-in-a-database sense.  It refers to immutability.  For example, adding an element to a vector creates a new vector that shares structure with the old one.  Because all data structures are immutable, this is both safe and efficient.  Clojure’s hash maps, for example, have time complexity of log-base-32, which is so small it’s practically constant.

• The first thing Rich did when experimenting with the semantic web was to pull data out of the Jena API and get it into Clojure data structures.  That allows him to leverage the full power of Clojure’s data manipulation functions.  This opens up a world of possibilities that he wouldn’t have if he stuck with Jena objects.  Basically, having your data trapped inside objects is bad, because you’re limited to whatever methods those objects provide.  With generic data structures, you can re-use and compose all the functions that Clojure already provides.

Screencasts and code from the talk should appear soon — watch clojure.org or the Clojure Google group for an announcement.

I dropped by the Java Users’ Group meeting last week since Rich Hickey was there to talk about Clojure.

I expected a bit of carping from the Java guys, and at first they were all “efficiency this” and “security that.”  But by mid-way through the talk I think they were getting it.  A few even got excited about macros.

If I didn’t make it clear in my first post about Clojure, I like this language.  Here’s some more reasons why:

1. All binding constructs — let, defn, and the like — perform destructuring.
2. Universal data structures — vector, list, map, set.
3. Built-in java.util.regex support.
4. Sixty squintillion Java libraries.
5. A small but growing number of Clojure libraries, some written by yours truly.
6. You can generate Java .class files that run Clojure code.
7. Lightning-fast bug fixes from the author.

… as explained by Sir Kenny,

From: Ken Tilton
Newsgroups: comp.lang.lisp
Date: Tue, 01 Apr 2008 14:53:07 -0400
Subject: Re: Newbie FAQ #2: Where’s the GUI?

Jonathan Gardner wrote:
> I know this is a FAQ, but I still don’t have any answers, at least answers that I like.

That’s because you missed FAQ #1 (”Where are the damn libraries?”) and the answer (”The Open Source Fairy has left the building. Do them your own damn self.”)

… message truncated …

The most famous piece of Lisp-related vaporware is vapor no longer: Arc has been released. After paging through the tutorial, I’m a bit underwhelmed. It looks like just a bunch of syntactic sugar implemented on top of Scheme. Clojure is more interesting and more innovative. Clojure and Arc have some things in common: data structures are functions on their keys, strings are sequences, and lambda is “fn”. But Arc seems to be more in the traditional, semi-imperative model whereas Clojure borrows heavily from functional programming. Arc even has for-loops, for goodness sakes. Obviously, I have no idea what else Paul Graham may have up his sleeve, so maybe there’s much more exciting stuff to come. But for now Arc looks to me like a modest improvement on Common Lisp / Scheme rather than a major new language.

Dan Weinreb posted common Complaints About Common Lisp. My personal complaint is in there — the lack of libraries that are well-documented and regularly updated. I think it’s a critical mass problem: so few people are using Common Lisp in their day-to-day work that there’s not enough momentum to keep the libraries going and make them fool-proof. Too many Common Lisp libraries are weekend projects that never make it out of alpha.

I’m guilty of the same offense: my one and only (very small) Common Lisp library — a bridge to run an embedded Perl 5 interpreter from Common Lisp — went a year before I heard from one lone user. By that time I had switched to Ruby.

The test of a really good library is not that it’s there, but that you don’t notice it’s there. If I want to scrape some HTML in Ruby, I don’t need to think about it, I just type “require ‘hpricot’” and it works. If I have a problem, odds are someone else has had the same problem and Google will find it. With Common Lisp, one can feel like a lone voice crying out in the wilderness. There’s a bit of a frontier mentality, too: “Well, if it’s broke, fix it yourself, citizen.”

A couple nights ago I walked down to LispNYC in the East Village to hear Rich Hickey talk about Clojure, his new Lisp-like language. To be honest, I wasn’t expecting much. Another Lisp? Ho hum. I’m sure it’s very clever and cool and all, but not something I can actually use.

Instead, I was blown away by Rich’s presentation. Clojure might just be the Lisp I’ve been waiting for. Here’s why:

1. Clojure targets the JVM. That doesn’t just mean it’s written in Java. It compiles directly to Java bytecode, so it can take advantage of all the optimization work that’s been done on just-in-time bytecode compilers. So it’s both fast and portable.
2. Clojure is functional, i.e. “side-effect free.” The built-in data structures are immutable. At the same time, you can have side effects (via external Java calls) without needing monads.
3. Clojure has modern data structures. Vectors and hash maps are built in, and both can be used as “functions” that take a key value as their argument (I immediately thought yes, that’s the way it should be).
4. Clojure fits into the Java ecosystem. All squintillion Java library methods can be called directly from Clojure. (Hickey demonstrated a small Swing app with fewer lines than the Java version.) Clojure data structures implement the Java collection interfaces.
5. Clojure has built-in concurrency support. I barely understand this, but Hickey has done his homework and built a sophisticated concurrency and transaction system right into the language.
6. Clojure abstracts Lisp sequences. Standard functions like “map” and “filter” work on any sequence, not just lists. In fact, they work on any object that implements the Java “Iterable” interface.
7. Clojure has metadata. Objects can have metadata attached to them, which can be used by the program without affecting the value of the data (e.g. two objects with identical contents but different metadata are still considered “equal”).
8. Clojure is still Lisp! It has first-class functions, Common Lisp-style macros (with automatic gensyms), a reader, “eval”, and a REPL.

I haven’t been this excited about Lisp since Peter Seibel came to talk about Practical Common Lisp. Hickey’s full presentation should show up soon at LispNYC.org. There’s also a (beautiful) Clojure web site and a Google group.

Update Nov. 21: Audio and slides from the presentation are available at LispNYC.

From a 1995 paper on intentional programming: “Present day syntax had [sic] been predicated on a character stream that could have been input from punch cards, or teletypes.” Exactly! Why are we still working in a punch-card manner on million-pixel displays? Why are we still arguing about curly brackets versus parentheses when Unicode has millions of characters? In short, why do we have syntax at all?

Abandoning syntax wouldn’t mean abandoning “real” programming. Visual programming probably didn’t catch on because designing software with a mouse was too slow. But that’s a fault of the interface, not the method. There’s no reason why an tree-like editor in an Intentional Programming system couldn’t be as slick as Emacs with Paredit. Assuming the editor were also developed with Intentional Programming, it would be just as extendable as Emacs.

From the same paper: “Meta-work (i.e. consistent rewriting of a program) is not expressible in languages even though it comprises a substantial part of programmer’s workload. Lisp is an exception in this regard, but only at the cost of failing most other tests for usability.” I might argue with the usability statement, but the rest is true.

Sadly, Intentional Programming was originally developed at Microsoft, patented, and then seemingly abandoned. On the upside, the author of that paper now has his own company, Intentional Software, with patent deals with Microsoft and a blog.

I’m certainly not the first to do this, but I felt like writing it. Comparing Ruby and Common Lisp:

Syntax: Advantage, Common Lisp. No contest here. Ruby’s syntax is ugly, with all those ends hanging around and the { |var| ... } block syntax. The one thing Ruby has going for it is conciseness. The block syntax, ugly though it may be, is shorter than (lambda (var) ...), which may explain why Ruby uses blocks everywhere while CL programmers go out of their way to avoid lambdas.

Libraries: Advantage, Ruby. CL has some really interesting high-level libraries, but it’s lacking in bread-and-butter utilities like date/time, file handling, and string munging.

Speed: Everybody knows Ruby is slow. CL with native compilation is not. On the other hand, a lot of Ruby libraries are written in hand-optimized C, so they’re plenty fast. But just try to decipher that C code when you want to modify the behavior of a library. Slight advantage, CL.

Resource usage: Slight advantage, Ruby. Most CL implementations carry the baggage of a 20MB runtime. Ruby by itself is small, but some major libraries (e.g. Rails) are memory hogs.

Web development: Advantage, Ruby. Rails rocks. The CL web frameworks are complex and not well tested in mainstream production use.

Testing: Dead heat. Both languages have several excellent testing frameworks. There’s been some particularly innovative work on Behavior-Driven Development in Ruby with rSpec.

Metaprogramming: Advantage, CL. Although Ruby is famous for its metaprogramming abilities, it can’t compete with CL’s macro system and the Meta-Object Protocol. I find the Ruby metaprogramming methods confusing, so I fall back on evaluating template strings, which is error-prone.

Difficulty: Ruby is definitely easier to learn, especially for someone with some C/Perl background. Common Lisp is really different, and shows its age in areas like file handling.

Code organization: This can’t be quantified, but I find it easier to structure my code in Ruby. Knowing at the outset that everything will go into classes and methods makes it more obvious how code is divided up. Lisp code, the “big ball of mud” as some call it, grows more organically but leaves me with a disorganized mess of individual functions scattered across a bunch of source files. I think writing well-organized code in Common Lisp requires more discipline and attention to detail than it does in a class-oriented language like Ruby.

Conclusion: Both languages have their problems. I feel more affection for Common Lisp, and I’m glad I learned it, but Ruby will probably continue to be my primary working language for a while. If I were starting something really unique that I would have to build from the ground up, Lisp would be my choice. But for building dynamic web sites, Ruby gets the job done right now. I hope Ruby will continue to evolve in a Lispy direction, with an abstract syntax tree and a macro system. Throw in an optimizing compiler and it might almost be perfect.

I was at LispNYC last night listening to Anton van Straaten discuss his work on R6RS, the new Scheme standard. One surprising change from R5RS is that eval is defined in a library.

Eval, in a library? Holy scopes! The Common Lispers in the audience were aghast. Even the Schemers were a tad confused. Anton explained. The goal of Scheme, he said, has always been to incorporate as much dynamic behavior as possible without sacrificing efficient compilation. Towards this end, the R6RS eval is more limited than Common Lisp’s eval.

As I understand it, eval was central to McCarthy’s original design for LISP. Eval is LISP, and LISP is eval. Of course, as others reminded me later, LISP’s definition of eval with dynamic scope led to the 30-year “funarg” bug. Eval is also a thorn in the side of anyone trying to generate a standalone Common Lisp program — the possibility of a call to eval means the compiler has to include the entire Common Lisp runtime (up to 20 MB, depending on the implementation) in the final executable.

This got me thinking about Ruby, too. While Common Lisp and Scheme actually discourage the use of eval, Ruby is pretty casual about it. In a purely interpreted, dynamic language, that’s not a problem. But it would make it tough to implement a static Ruby compiler.

In one of my first posts, I asked “Why do we speak of programming languages instead of programming notation?” My thought was, and still is, that code in any existing programming language is just one possible representation of an abstract computational process. Higher-level languages like Lisp are good because they bring the written representation closer to the abstraction. But are programming languages necessary at all?

Perhaps not: some smart people are working on an idea I’ve toyed with myself, but lacked the experience and theoretical background to tackle. They call it Intentional Programming, and also use the word “notation.” Basically, all your “code” becomes part of an abstract syntax tree stored in a versioned database. You use visual and automated tools to manipulate that tree, then a generator “compiles” your tree into source code in a language of your choice.

Dave Roberts compares this to the Lisp way of things, in which the abstract syntax tree and the textual representation are nearly identical.

Some more Intentional Programming links here.

The link to the Microsoft demo video no longer works; if anyone has a substitute, please send it my way.