Common Lisp the Language (CLtL) devotes an entire chapter to the subject of Scope and Extent. It defines scope as the textual region of a program in which an entity may be used, where “entity” could be a symbol, a value, or something more abstract like a variable binding.
So scope is about where you can use something. CLtL defines two different kinds of scope:
Lexical scope is usually the body of a single expression like let or defun.
Indefinite scope is everything else, effectively the global set of symbols that exist in a program.
In contrast, extent is about when: it’s the interval of time during which something may be used. CLtL defines two different kinds of extent:
Dynamic extent refers to things that exist for a fixed period of time and are explicitly “destroyed” at the end of that period, usually when control returns to the code that created the thing.
Indefinite extent is everything else: things that get created, passed around, and eventually garbage-collected.
In any language with a garbage collector, most things have indefinite extent. You can create strings, lists, or hash tables and pass them around with impunity. When the garbage collector determines that you are done using something, it reclaims the memory. The process is, in most cases, completely transparent to you.
But what about the so-called dynamic scope? The authors of CLtL have this to say:
The term “dynamic scope” is a misnomer. Nevertheless it is both traditional and useful.
They also define “dynamic scope” to be the combination of indefinite scope and dynamic extent. That is, things with dynamic scope are valid in any place in a program, but only for a limited time. In Common Lisp, these are called “special variables,” and are created with the macros defparameter and defvar.
Vars
So what does this have to do with Clojure? Clojure has these things called Vars. Every time you write def or defn or one of its variants in Clojure, you’re creating a Var.
Vars have indefinite scope: no matter where you def a Var, it’s visible everywhere in the program.1
Vars usually have indefinite extent as well. Usually. This is where things get tricky. Clojure, unlike Common Lisp, was designed for multi-threaded programs.2 The meaning of extent gets a lot muddier in the face of multiple threads. Each thread has its own timeline, its own view of “now” which may or may not conform to any other thread’s view.
In Clojure versions 1.2 and earlier, all Vars had dynamic scope by default, but this meant that there was a performance cost to look up the current dynamic binding of a Var on every function call. Leading up to 1.3, Rich Hickey experimented with allowing Vars to be declared ^:static, before settling on static by default with ^:dynamic as an option. You can still find ^:static declarations littered through the Clojure source code. Maybe someday they’ll be useful again.
The definition of “dynamic scope” in Clojure is even fuzzier than it is in Common Lisp. How do we define “extent” in the face of multiple threads, each potentially with its own thread-local binding? If a resource can be shared across multiple threads, we have to coordinate the cleanup of that resource. For example, if I open a socket and then hand it off to another piece of code, who is responsible for closing the socket?
Resource management is one concurrency bugaboo that Clojure developers have not managed to crack. Various attempts have been made: you can see the artifacts in wiki and mailing list discussions of Resource Scopes. So far, no solution has been found that doesn’t just shift the problem somewhere else.
It ends up looking a bit like garbage collection: how do I track the path of every resource used by my program and ensure that it gets cleaned up at the appropriate time? But it’s even harder than that, because resources like file handles and sockets are much scarcer than memory: they need to be reclaimed as soon as possible. In a modern runtime like the JVM, garbage collection is stochastic: there’s no guarantee that it will happen at any particular time, or even that it will happen at all.
To make matters worse, Clojure has laziness to contend with. It’s entirely possible to obtain a resource, start consuming it via a lazy sequence, and never finish consuming it.
The Wrong Solution
This brings me to one of my top anti-patterns in Clojure: the Dynamically-Scoped Singleton Resource (DSSR).
The DSSR is popular in libraries that depend on some external resource such as a socket, file, or database connection. It typically looks like this:
(ns com.example.library) (def ^:dynamic *resource*) (defn- internal-procedure [] ;; ... uses *resource* ... ) (defn public-api-function [arg] ;; ... calls internal-procedure ... )
That is, there is a single dynamic Var holding the “resource” on which the rest of the API operates. The DSSR is often accompanied by a with-* macro:
(defmacro with-resource [src & body] `(binding [*resource* (acquire src)] (try ~@body (finally (dispose *resource*)))))
This looks harmless enough. It’s practically a carbon copy of Clojure’s with-open macro, and it ensures that the resource will get cleaned up even if body throws an exception.
The problem with this pattern, especially in libraries, is the constraints it imposes on any code that wants to use the library. The with-resource macro severely constrains what you can do in the body:
You can’t dispatch to another thread. Say goodbye to Agents, Futures, thread pools, non-blocking I/O, or any other kind of asynchrony. The resource is only valid on the current thread.3
You can’t return a lazy sequence backed by the resource because the resource will be destroyed as soon as body returns.
You can’t have more than one resource at a time. Hence the “singleton” in the name of this pattern. Using a thread-bound Var throughout the API means that you can never operate on more than one instance of the resource in a single thread. Lots of apps need to work with multiple databases, which really sucks using this kind of library.
The last problem with this pattern is a more subtle one: hidden dependencies. The public API functions, which have global scope, depend on the state (thread-local binding) of another Var with global scope. This dependency isn’t explicitly stated anywhere in the definition of those functions. That might not seem like such a big deal in small examples, and it isn’t. But as programs (and development teams) grow larger, it’s one additional piece of implicit knowledge that you have to keep in your head. If there are seventeen layers of function calls between the resource binding and its usage, how certain are you going to be that the resource has the right extent?
Friends Don’t Let Friends Use Dynamic Scope
The alternative is easy: don’t do it. Don’t try to “solve” resource management in every library.
By all means, provide the functions to acquire and dispose of resources, but then let the application programmer decide what to do with them. Define API functions to take the resource as an argument.
Applications can manage their own resources, and only the application programmer knows what the extent of those resources should be. Maybe you can pass it around as a value. Maybe you want to use dynamic binding after all. Maybe you want to stash it in a global state Var.4 That’s for you to decide.
Datomic is a good example to follow: it creates connection objects that have a lot of state attached to them — sockets, queues, and threads. But it says nothing about how you should manage the extent of those connections.5 Most functions in the Datomic API take either a connection or a database (a value obtained from the connection) as an argument.
Safe Dynamic Scope
So dynamic scope is totally evil, right? Not totally. There are situations where dynamic scope can be helpful without causing the cascade of problems I described above.
Remember that dynamic scope in Clojure is really thread-local binding. Therefore, it’s best suited to operations that are confined to a single thread. There are plenty of examples of this: most popular algorithms are single-threaded, after all. Consider the classic recursive-descent parser: you start with one function call at the top and you’re not done until that function returns. The entire operation happens on a single thread, in a single call stack. It has dynamic extent.
I took advantage of this fact in a Clojure JSON parser. There were a number of control flags that I needed to make available to all the functions. Rather than pass around extra arguments all over the place, I created private dynamic Vars to hold them. Those Vars get bound in the entry-point to the parser, based on options passed in as arguments to the public API function. The thread-local state never leaks out of the initial function call.
As another example, the Clojure compiler, although written in Java, uses dynamic Vars to keep track of internal state.
And what about our friend with-open? I said that the example with-resource macro was nearly a copy of it, but only nearly. clojure.core/with-open creates lexical (i.e. local) bindings. It still suffers from some limitations around what you can do in the body, but at least it doesn’t limit you to one resource at a time.
Global state is the zombie in the closet of every Clojure program, about which I’ll have more to say in future posts. For now, I hope I’ve convinced you that dynamic scope is easily abused and has a lot of unintended consequences.
Footnotes:
1 Technically, a Var is visible only after the point at which it was defined. This is significant with regard to the order of definitions, but Vars are still globally visible once they have been defined.
2 CLtL has this note embedded in the chapter on scope and extent: “Behind the assertion that dynamic extents nest properly is the assumption that there is only a single program or process. Common Lisp does not address the problems of multiprogramming (timesharing) or multiprocessing (more than one active processor) within a single Lisp environment.” Modern Common Lisp implementations have added multi-threading, but it remains absent from the language specification.
3 You can use bound-fn to capture the bindings and pass them to another thread, but you still have the problem that the resource may be destroyed before the other thread is finished with it.
4 Not recommended, to be discussed in a future post.
5 There’s some caching of connection objects under the hood, but this is not relevant to the consumer.
One point I don’t see here is the painful interactions between dynamic scope and lazyness. Like you mention for parsing, I’ved dynamic scope for passing around configuration information while creating tree like data structures, for example a webpage DOM. What I always forget, however, is that bound variables are not kept with the environment. A call to
mapwill create a lazy sequence that will contain the lexical scope, but not the dynamic scope. When this seq is evaluated outside of the initial creation call stack, the bindings are lost. I’ve been bitten by this many times. Consequently, I’ve given up on lazyness in Clojure: it’s more of a headache than a feature for the kind of code I write.I take it when you say “You can’t have more than one resource at a time”, you’re not including the case where you can nest such `with-*` macros, i.e.,
(def ^:dynamic *resource* nil)
(with-resource res1
(println :resource *resource*)
(with-resource res2
(println :resource *resource*))
(println :resource "is back to" *resource*))
The use case for this pattern (or anti-pattern as it were) is one where there’s a thread local binding for a resource that needs to be properly released. It’s the create/release mechanism that the library is abstracting away. I’m not entirely sure why this is a bad thing, so long as the user understands the other tradeoffs related to thread boundaries and laziness.
Your point about providing access to to the create/release mechanism (in addition to the `with-*`) is valuable. And, if the caller decides to take complete control of the resource management, at least the `with-*` serves as a reasonable documentation for what needs to be done.
But, I worry about a statement like:
“The alternative is easy: don’t do it. Don’t try to “solve” resource management in every library”
because there are some who will take this as gospel, and decide that this kind of abstraction needs to be done away with altogether.
Hi Ram,
Even when you nest different bindings of
*resource*, you’re still limited by the API to using one at a time. You cannot interleave API calls targeting two different resources without adding a lot of extra code to swap the bindings at every step.“there are some who will … decide that this kind of abstraction needs to be done away with altogether.”
That is my hope. :)
-S
Great article. This whole point (the problems with dynamic scope) is part of what inspired me to start thinking about “The Environment as a Value” – ref my post on the clojure-dev list: https://groups.google.com/d/topic/clojure-dev/S8BawG7nzJA/discussion
The other solution I’ve found to this is to use lots of closures (effectively embedding scope within functions) and compose them with HOFs. This works and is nicely functional, although it can get a bit messy and painful when you need to thread some new parameter through an existing call chain.
You are 100% right!
I too have thing against dynamic vars, it always felt impure (not ot mention less performant + all other problems you mentioned) and more of a burden than anything, when it can simply be replaced (in most cases) by function arguments.
I have yet to see one real world use case of deep nesting of with- macros that would really improve the code/readability too.
As you said there are some use cases that are valid, but the sad part is the fact that it is advertised as “idiomatic” by a lot of people and some users will complain if you don’t provide that since it’s expected now…
An (sadly) some very good libraries rely on that as well. I just spent a good part of the day porting some code to one of these, the quality and functionalities are excellent, but you spend your time fighting these macros and end up dreaming about a simple arg+partial version of it.
As a result I tend to provide both styles when I can, but it still feels like a stain in the code to be honest.
Committing this sin is very convenient. The best way I’ve found to avoid this pattern is to inject dependencies/resources into a record or type that exposes the API (i think you also gave a related talk). I think most people don’t do this though because it’s considered too close to OO (?).
Out of curiosity – in your mind is having two versions of functions (one that uses the default resource and one that allows passing it in) just as gross?
btw, clojure now does magic to propagate dynamic values to futures? or did I misunderstand above?
user> (def ^:dynamic *x*)
#’user/*x*
user> (defmacro with-x [v & body] `(binding [*x* ~v] (do ~@body)))
#’user/with-x
user> (with-x 3 (future (Thread/sleep 2000) (println *x*)))
#
3
Another solution is to use kw args defaults, but that’s not usable for leading argument(s):
(defn foo [x y }]
...)
About futures yes, there is a binding-conveyor-fn that does the magic: https://github.com/clojure/clojure/blob/master/src/clj/clojure/core.clj#L1828
My code was cut in the middle, here is what I meant: https://www.refheap.com/paste/13144
Hi Ignacio,
Yes, it is convenient. That’s part of the problem. :)
“in your mind is having two versions of functions (one that uses the default resource and one that allows passing it in) just as gross?”
I try to stay away from emotion-laden words like “gross,” but I do think that trying to provide multiple versions of the same API is more trouble than it’s worth. You’ll inevitably try some “clever” code tricks to avoid duplication, which are more likely to cause problems down the road.
Instead, focus on making your API as simple as possible (see Rich Hickey’s talks on Simple versus Easy) then leave it up to consumers to add additional conveniences that suit their specific use cases.
And yes, in recent versions of Clojure, dynamic bindings are “conveyed” to other threads. In Clojure 1.3 the implementation had a bug that caused a pretty serious memory leak. This was fixed in later releases. Even with binding conveyance, though, you still have the resource-extent and singleton problems I described.
-S
Great article Stuart, I have one question though. I am just learning Clojure so pardon me if this is a dumb question. What is your recommendation for long-lived code that needs to maintain some state internally that spans function calls? An example might be some metric information about the service calls being made or some summary of the results returned on an API across all it’s clients. In this case the state is long-lived and not tied to a specific function invocation. Assuming that we’ve dealt with the multi-threading issues by some other means (STM,agents,refs,etc) , what’s the best way to declare/bind said vars?
I see this a more general problem because unlike parsing (which is a purely functional transformation), there is context “left over” here after a client call to the API function(s) that will be used in subsequent calls. I know that the general recommendation is to avoid building an API that requires these semantics as much as possible, but I just don’t know else you would do it. Thoughts?
Hi Cliff,
You can still support long-lived state with this technique. Each function can still take the state (ref, atom, agent) as a parameter and operate on it.
As to where you keep the state between function calls, I hope to write more about it in the future. My general approach is to create an “instance” of my application which encapsulates all of its state, then pass that state around through explicit function arguments or closures. I not to have any state in global Vars, but sometimes deployment environments make that impossible to avoid.