In the early days of Clojure, I was skeptical of Clojure-specific build tools like Lancet, Leiningen, and Cake. Why would Clojure, a part of the Java ecosystem, need its own build tool when there were already so many Java-based tools?
At the time, I thought Maven was the last word in build tooling. Early Leiningen felt like a thin wrapper around Maven for people with an inconsolable allergy to XML. Maven was the serious build tool, with a rich declarative model for describing dependency relationships among software artifacts. That model was imperfect, but it worked well enough to power one of the largest repositories of open-source software on the planet.
But things change. Leiningen has evolved rapidly. Maven has also evolved, but more slowly, and the promised non-XML POM syntax (“polyglot Maven”) has not materialized.
Meanwhile, I learned why everyone eventually hates Maven, through the experience of crafting custom Maven builds for two large-ish projects: the Clojure language and its contributed libraries. It was a challenge to satisfy the (often conflicting) requirements of developers, continuous integration, source repositories, and the public Maven repository network. Even with the help of Maven books from Sonatype, it took months of trial and error and nearly all my “open-source” time to get everything working.
At the end of this process I discovered, to my dismay, that I was the only one who understood it. As my colleague Stuart Halloway put it, “Maven breeds heroes.” For end-users and developers, there’s a nice interface: Clojure-contrib library authors can literally click a button to make a release. But behind that button are so many steps and moving parts (Git, Hudson, Maven, Nexus, GPG, and all the Maven plugins) that even I can barely remember how it all works. I never wanted to be the XML hero.
So I have come around to Leiningen, and even incorporate it into my Clojure development workflow. It’s had some bumps, as one might expect from a fast-moving open-source project with lots of contributors, but most of the time it does what I need and doesn’t get in the way.
What puzzles me, however, is the stubbornness of developers who want to do everything via Leiningen. Some days it seems like every new tool or development utility for Clojure comes wrapped up in a Leiningen plugin so it can be invoked at the command line. I don’t get it. When you have a Clojure REPL, why would you limit yourself to the UNIX shell?
I think this habit comes partly from scripting languages, which were born at the command line, and still live there to a great extent. But it puzzled me a bit even in Ruby: if it takes 3 seconds to for rake to load your 5000-line Rails app, do you really want to use rake for critical administrative tasks like database migrations? IRB is not a REPL in the Lisp sense, but it’s a pretty good interactive shell. I’d rather work with a large Ruby app in IRB than via rake.
Start-up time remains a major concern for Leiningen, and its contributors have gone to great lengths (sometimes too far) to ameliorate it. Why not just avoid the problem altogether? Start Leiningen once and then work at the REPL. Admittedly, this takes some discipline and careful application design, but on my own projects I’ve gotten to the point where I only need to launch Leiningen once a day. Occasionally I make a mistake and get my application into such a borked state that the only remedy is restarting the JVM, but those occasions are rare.
I pretty much use Leiningen for just three things: 1) getting dependencies, 2) building JARs, and 3) launching REPLs. Once I have a REPL I can do my real work: running my application, testing, and profiling. The feedback cycles are faster and the debugging options much richer than what I can get on the command-line.
“Build plugins,” for Leiningen or Maven or any other tool, always suffer from running in a different environment from the code they are building. But isn’t one of the central tenets of Lisp that the compiler is part of your application? There isn’t really a sharp boundary between “build” code and “application” code. It’s all just code.
I used to write little “command-line interfaces” for running tests, builds, deployments, and so on. Now I’m more likely to just put those functions in a Clojure namespace and call them from the REPL. Sometimes I wonder: why not go further? Use Leiningen (or Maven, or Gradle, or whatever) just to download dependencies and bootstrap a REPL, then execute builds and releases from the REPL.
Hm, can’t say I agree.
I think there are several very good reasons why your application’s main “entry points” (including tests) should be accessible via the command line:
1. It’s easier to run them in CI.
2. Its easier to script execution in a DevOps scenario
3. Your program can be executed by people who don’t know Clojure (or who aren’t even developers)
4. It makes the app more self-documenting about what the primary entry points are
I agree to the extent that it makes sense to use the REPL *while developing* a project, but I do think the projects external API should be exposed to command-line users, and lein is a straightforward way to do that.
Luke,
Totally agree that those are good reasons to have command-line entry points. I just don’t want the command-line to be the only entry point.
I’m not a big Clojure user, but I too have been puzzled by repeatedly launching Leiningen, rather than entering a REPL and staying there. One thing that drew me to moving from Maven to SBT (even before I actually moved from Java to Scala for development in general) was that I can type “sbt” and then do stuff from the SBT console and stay there. Having stuff preloaded, and using SBT’s incremental compilation and testing and other features, has been fairly pleasant.
Why not just do up a little drop-in to tools.cli that emulates the -f (execute function) -l (load code from file), and –eval. It seems like that would give you the best of both worlds.
on a related topic, I wish I had a clojure shell I could use instead of a unix shell, this would be the place for everyday unix commands, like cd, grep and find. if clojure-function versions of those returned data structures instead of text streams, then piping stuff would be easier.
I have recently heard of eshell which may be something similar to what I’m looking for, havent had a chance to sit down with it yet.
People writing Leiningen plugins when they shouldn’t is actually a really common problem. The best way to solve this problem is to write a `-main` function which you can invoke from the repl, and then if you want to expose it from the command line, write an alias that partially applies the `run` task like so:
:aliases {“go” [“run” “-m” “myapp.start”]}
This will make `lein go fast` eval `(myapp.start/-main “fast”)` inside the context of your project.
I should probably get around to blogging about this since it’s not widely understood. There are still valid reasons to write a lein plugin if you need access to the project map for some reason, but people using it as a thin wrapper around `eval-in-project` need to cut that out.
> Use Leiningen (or Maven, or Gradle, or whatever) just to download dependencies and bootstrap a REPL, then execute builds and releases from the REPL.
This is an interesting idea. I would certainly like to do this with the test task; (once lein 3.0 rolls around) the existing implementation is basically a textbook example of the “putting too much in a plugin” problem I mentioned earlier; because tests run in the project process it should definitely be a library. The problem is it’s basically just a big monkeypatch on clojure.test.
I don’t think it’s as clear-cut for the jar or deploy tasks, especially since the deploy task brings in a nontrivial pile of deps which are likely to cause conflicts with existing project deps; JVM isolation in this case is actually quite valuable. But if you had unique nontrivial packaging requirements it could make sense in some cases.