A. Debugging Macros
The goal of this appendix is to provide you with techniques, tools, and a mental model to apply when examining unexpected macro behavior. If your macros are working fine, you can skip this one!
Assumptions
In this Appendix we assume the following:
You’re working through a problem that requires macros and you’re trying to understand what the program is doing.
You’re comfortable with using the Clojure REPL to analyze a smaller part of a larger project, and solve the problem using this tool.
Benefits
The benefit of working through this Appendix is that when you can’t wrap your head around what a macro is doing, you’ll have a set of steps to see the phases of a macro and the ability to peer inside each of them.
The Recipe
In this Appendix we’ll start small and build up by looking at:
A simple approach
Some helper functions
A mental model for the read and evaluate phases
Some functions to look at the read phase
Some functions to look at the evaluate phase
A Simple Approach—Expansion-Time and Evaluation-Time stdout
On a simple level, we can break everything that happens in Clojure macros into expansion time and evaluation time.
Expansion time—When the code template inside the macro is expanded, for example, the macro is evaluated.
Evaluation time—When the resulting macro-generated code is evaluated.
Suppose we have the following macro to generate a function:
(defmacro adder-maker [x]
`(defn adder [c# d#]
(+ c# d#)))
And we use it like this:
user=> (adder-maker 1)
#adder
user=> (adder 3 4)
7
Now suppose we were stuck in a situation where we didn’t know what the arguments to the macro were (from the get-args function, which might have come from another library):
user=> (adder-maker (get-args))
#user/adder
We’ll mock up the get-args function here but pretend for the purposes of debugging that we don’t know what values it is returning due to encapsulation:
(defn get-args [] '(42 123))
We could add a simple println statement:
(defmacro adder-maker [x]
(println x)
(println (eval x))
(clojure.lang.RT/var "myns" "adder"
(fn [c d]
(+ c d))))
The first thing to note is that this function call would get evaluated at expansion time because it is not inside a backtick.
user=> (adder-maker (get-args))
(get-args)
(42 123)
You can see that the results are generated at the time the macro is evaluated, i.e., expansion time for the code template inside the macro. But if we wanted to test inside the macro’s template code, we’d have to put it inside the function like so:
(defmacro adder-maker [x]
`(defn adder [c# d#]
(println c# d#)
(+ c# d#)))
user=> (adder-maker (get-args))
#user/adder
user=> (adder 3 4)
3 4
7
You can see we didn’t get results until after the code that was inside the macro template was expanded and then evaluated.
What if we wanted to print the macro arguments at evaluation time? Then we’d need to quote them, like this:
(defmacro adder-maker [x]
`(defn adder [c# d#]
(println (eval '~x))
(+ c# d#)))
user=> (adder-maker (get-args))
#user/adder
user=> (adder 3 4)
(42 123)
7
You can see that the original arguments from the get-args function are now being displayed after compilation, when the code that was inside the macro template is now being evaluated.
Some Macro Helper Functions
Here are some specialized tools to use when debugging macros. You might add them to your Clojure Swiss Army knife. The first is macroexpand-1 (which is itself a macro.) This lets us expand the macro in question to one level. First we’ll re-create our original macro:
(defmacro adder-maker [x]
`(defn adder [c# d#]
(+ c# d#)))
Now suppose we had our macro being called like the following:
(adder-maker (get-args))
Now suppose we didn’t know what the function get-args was returning, or what the macro adder-maker was expanding to. To get more information, we could use our new tool, as in the following:
user=>(macroexpand-1 `(adder-maker (get-args)))
(clojure.core/defn user/adder [c__1448__auto__ d__1449__auto__] (clojure.core/+
c__1448__auto__ d__1449__auto__))
This tells us that the arguments that get passed in don’t actually get used because they aren’t referenced in the output of macroexpand-1.
Now let’s look at another tool, macroexpand. So what’s the difference between macroexpand-1 and macroexpand? Good question! In general, we choose the best tool for the job. You’d use macroexpand when you have multiple macros chained together and want to see the final result, ignoring the expansion steps. With macroexpand-1 you could expand these chained macros just one step.
Let’s apply this. Suppose we have a macro adder-maker-wrapper that refers to our previous macro adder-maker:
(defmacro adder-maker [x]
`(defn adder [c# d#]
(+ c# d#)))
(defmacro adder-maker-wrapper [e f]
`(adder-maker e))
We call it with macroexpand-1:
user=>(macroexpand-1 `(adder-maker-wrapper 1 2))
(user/adder-maker user/e user/f)
You can see this has unwrapped the macros just one level. When we call it with macroexpand:
user=>(macroexpand `(adder-maker-wrapper 1 2))
(def user/adder (clojure.core/fn ([c__1457__auto__ d__1458__auto__] (clojure.
core/+ c__1457__auto__ d__1458__auto__))))
you can see that this has expanded the two chained macros into our final adder function.
Read and Evaluate—A More Developed Mental Model
The previous techniques are OK for simpler problems, but sometimes we want to deepen our mental model of what is going on. When looking at macros, we fall back to the model of how the REPL works. There are four phases: read, evaluate, print, and loop.
We can illustrate these phases with Table A.1 by populating some Clojure functions that correspond to the phases we’re interested in: read and evaluate.
Reading
In Clojure, reading input from the REPL is done in a module called the reader. We can look at what the reader is doing by using the read-string function. For example:
user=> (read-string "'a")
(quote a)
Here we see that the string has been turned into a Clojure object, but it hasn’t yet been evaluated. We can examine this further to get more information about it by using the class function:
user=> (class (read-string "'a"))
clojure.lang.Cons
Here we see the class of the Clojure object that has not yet been evaluated. We can use this technique to break up a part of our macro and see what it would do. Given the following macro:
(defmacro a []
`b#)
If we’re curious about the auto-gensym b#, we could evaluate it:
user=> (read-string "`b#")
=> (quote b__14__auto__)
Here the gensym is expanded into a reference variable that would remain constant throughout the macro expansion but would generally remain invisible to the end user. Note that you might get a slightly different result for a gensym, but we expect you’d get a result that looked quite similar to what we have here. We’ll repeat this with another example because understanding how the Clojure reader works is quite important. Given the following macro:
(defmacro simple-adder []
`(#(+ 1 %)))
to investigate what the body of the macro template is doing we could do:
user=> (read-string "#(+ 1 %))")
(fn* [p1__17#] (+ 1 p1__17#))
By starting with the fn*, we see that the body of the macro is being expanded into a function. This comes from the expansion of the function # reader macro. Next we see that the function argument marked by % has been expanded into a gensym called p1__17#. Then we see that the body of the function has been expanded to add 1 to the parameter.
Another technique we’ll investigate in more detail with evaluation is unquoting a quoted parameter using the unquote ~ reader macro:
user=> (read-string "~e")
(clojure.core/unquote e)
You can see that this evaluates to the unquote function. We can then chain this with the quote function to delay execution of the unquote if necessary.
user=> (read-string "'~e")
(quote (clojure.core/unquote e))
Mixed with the eval function, this will be a tool to delay evaluation when examining a part of a macro.
Next we’ll reverse the technique and unquote a quoted function:
user=> (read-string "~'e")
(clojure.core/unquote (quote e))
In evaluation, this will enable us to remove a part of the macro that is quoted.
In summary, debugging macros at the read phase requires understanding these steps:
The Clojure reader turns strings into Clojure objects.
Use the read-string function to simulate this phase of macro expansion.
Use the class function to identify which Clojure object the reader has turned the string into.
Extract the body of a macro including the backquote into a string argument for read-string to see what objects are returned.
If a macro contains a quoted form and you have no other option, use unquote to remove it.
If a macro returns a form that you wish to delay reading, then you can wrap that result in a quote (even if it is an unquote you’re wrapping).
Evaluating
Everything we’ve looked at so far with read-string has been going from strings to unevaluated Clojure objects. This is useful when you’re trying to work out what a particular combination of reader macros is doing. But in a macro you’re really interested in what the backquote (`) is doing. For that you need the eval function.
Remember that function arguments are evaluated and macro arguments are not. This means that function arguments need to be valid Clojure forms, but macro arguments do not. The classic example is the unquoted ASCII character a. You could pass this as an argument to a macro as is but you’d need to quote it or put it in a list to pass it to a function.
We can summarize these differences in Table A.2.
The other explanation we need to put in here is the difference between quoting and backquotes. With a regular quote we are telling the evaluator that this is data, not code. With a backquote we are telling the evaluator to produce code that matches the form. That is, the backquote tells the evaluator the code it expects as a result of the expansion.
We can demonstrate this on the Clojure REPL:
user=> (read-string "'(* 2 3)")
(quote (* 2 3))
We can see that the reader macro for the quote function has been read into the Clojure quote object, but it has not been evaluated.
user=> (read-string "`(* 2 3)")
(clojure.core/seq (clojure.core/concat (clojure.core/list (quote clojure.core/*))
(clojure.core/list 2) (clojure.core/list 3)))
We can see that the code to produce the multiplication has been provided, as we can see the underlying Clojure objects.
We can summarize this idea as follows:
Expansion-time Quoting '—Telling the evaluator that this is data, not code.
Evaluation-time Backquoting `—Telling the evaluator to produce code that matches this form.
Let’s pull all the ideas we’ve worked through together. We are going to work through the evaluate phase of debugging a macro, so let’s do some evaluating!
We’ll start with the most basic example, declaring a symbol by quoting it:
user=> (eval (read-string "'g"))
g
The end result is that the evaluation of the quote function gave us a new symbol g. Let’s do that again with a simple multiplication function:
user=> (eval (read-string "'(* 2 3)"))
(* 2 3)
We can see that the quote reader macro in front of the function told the evaluator to treat the form as data, not code, so we got an unevaluated list as a result. Now let’s evaluate a backquoted symbol reference.
user=> (eval (read-string "`(* 2 3)"))
(clojure.core/* 2 3)
We see that the function to produce the code has been returned. Now let’s look at the generation of gensyms inside a backquote:
user=> (eval (read-string "`(h# h#)"))
(h__79__auto__ h__79__auto__)
We can see that the two gensyms have been expanded into the same symbol inside the backquoted form. Now let’s combine the eval function, the backquote, and the unquote reader macro:
user=> (eval (read-string "`(~i)"))
CompilerException java.lang.RuntimeException: Unable to resolve symbol: i in this
context, compiling:(NO_SOURCE_PATH:34)
We get an undefined symbol. If we define the symbol and run it again:
user=> (def i 42)
#'user/i
user=> (eval (read-string "`(~i)"))
(42)
Then we see that the unquote reader macro isn’t expanded with the rest of the backquoted form, but instead is evaluated to its symbol at expansion time.
Note that you can’t use quotes inside a backquote to delay the operation of the unquote:
user=> (eval (read-string "`('~j)"))
CompilerException java.lang.RuntimeException: Unable to resolve symbol: j in this
context, compiling:(NO_SOURCE_PATH:37)
Note that you can use unquote to cancel a quote:
user=> (eval (read-string "`(~'a)"))
(a)
Here are the steps to keep in mind when your macro is going through the eval phase:
Debugging macros at the evaluate phase eval executes the Clojure objects returned by the reader (or read-string).
The arguments to a macro aren’t evaluated, unlike the arguments to a function.
Quoting a form tells the evaluation that it is data, not code.
Backquoting a form tells the evaluation to produce the code described in the form.
Combining the evaluator and reader allows you to examine the body of a macro with eval and read-line.
You can test that inside a backquoted form, gensyms should evaluate to the same generated symbol name.
When unquoting a symbol inside a backquoted form, if you haven’t declared the symbol, then your macro expansion will fail.
You can’t use quotes inside a backquote to delay the operation of the unquote.
You can use unquote to out a quote.
Conclusion
In this Appendix on debugging macros we have covered:
A simple approach—expansion-time and evaluation-time stdout.
Some macro helper functions.