10. Extending the Compiler with a Macro

Building on what we learned about macros in the previous chapter, we’ll now look at an example of a macro that converts blocking code to non-blocking code.

You know how to use Leiningen to create and run applications.

You are receptive to the idea that macros are easily overused and should be used sparingly.

You have read through the Appendix on Debugging Macros.

they don’t automatically evaluate their arguments and

macros can evaluate their contents at macro-expansion time or at runtime.

1. Create a new Leiningen project async-macro in your projects directory, and then change to that directory:

lein new app async-macro
cd async-macro

2. In the file src/async_macro/core.clj, enter the following:

Click here to view code image

(ns async-macro.core
  (:gen-class))

(defmacro go-off-main-thread-rotating [& args]
  (let [instructions (zipmap (range (count args)) args)
               size (count args)]
        (future
`(do
        ~(loop [x 0] (when (<= x 10)
          (let [y (mod x size)]
               (eval (get instructions y)))
               (Thread/sleep 100)
               (recur (inc x))))))))

(go-off-main-thread-rotating
  (println "a")
  (println "b")
  (println "c")
  (println "d"))

(def q (ref (clojure.lang.PersistentQueue/EMPTY)))

(dosync (alter q conj 1 2 3))

(defn qpop [queue-ref]
  (dosync
    (let [item (peek @queue-ref)]
      (alter queue-ref pop)
      item)))

(println "pop: " (qpop q))
(println "pop: " (qpop q))

(defmacro wake-on-queue-event [queue & args]
  (future
       (do
        (loop [x 0]
             (let [item  (qpop (eval queue))]
               (when item
                     (println "pop-wake: " item)
                     (mapcat eval args))); run the instructions passed in
when we get a queue item
             (Thread/sleep 100)
             (recur (inc x))))))

(wake-on-queue-event q (println "awake!"))

(dosync (alter q conj 4 5 6))

(defn -main
  "I don't do a whole lot ... yet."
  [& args]
  (future (Thread/sleep 2000)
       (println "Done")
       (System/exit 0))
  (shutdown-agents))

1. You can run the recipe with this:

lein run

2. This will give the following results:

a
pop:  1
pop:  2
pop-wake:  3
awake!
b
pop-wake:  4
awake!
c
pop-wake:  5
awake!
d
pop-wake:  6
awake!
a
b
c
d
a
b
c
Done

This output reflects code that is running multiple threads, so it is a little hard to understand at first.

The first part to notice is the a, b, c looping in the background:

a
b
c

We created a future that would take 10 iterations 100 ms apart and chose a println function call from a sequence of s-expressions. This runs in the background. (After this, ignore the a, b, c and look at the other parts.)

The next part to look at is the

pop:  1
pop:  2

Here we’re popping two values off a queue to test our qpop function.

The next thing to notice is the

pop-wake:  3
awake!

repeated for 4, 5, and 6. Here we are creating a macro that runs in the background, peeking into the queue and running a given s-expression when a value is on the queue. It “wakes up” when a value comes in on the queue.

Now the bigger idea here is that we can use macros to convert our blocking-operations to non-blocking code.

(ns async-macro.core
  (:gen-class))

The next thing to notice is a macro that takes sequence of s-expressions and runs 10 of them off the main thread, asynchronously:

Click here to view code image

(defmacro go-off-main-thread-rotating [& args]
  (let [instructions (zipmap (range (count args)) args)
          size (count args)]
  (future
  `(do
  ~(loop [x 0] (when (<= x 10)
    (let [y (mod x size)]
          (eval (get instructions y)))
          (Thread/sleep 100)
          (recur (inc x))))))))

(go-off-main-thread-rotating
  (println "a")
  (println "b")
  (println "c")
  (println "d"))

This is what causes our

a
b
c

running in the background in the middle of everything else. Notice as well the line

       ~(loop [x 0] (when (<= x 10)

It means this only chooses 10 s-expressions to run so we end up finishing at c. The main thing to observe about this is that the macro lets us treat code as data. The macro then picks an s-expression and eval’s it, and recurs.

On a simplistic level, this is just wrapping a future around the code to run in the background. On a more nuanced analysis, we’re breaking up a sequence of s-expressions to run whichever and whenever we like.

The benefit is that we’re converting a blocking operation (potentially a for-loop) with a non-blocking operation.

Now the next piece of code is the queue declaration:

Click here to view code image

(def q (ref (clojure.lang.PersistentQueue/EMPTY)))

(dosync (alter q conj 1 2 3))

We create an empty queue q and conj three integers onto it.

Next we define a function qpop to pop values off the queue, and then we pop the first two off it:

(defn qpop [queue-ref]
  (dosync
    (let [item (peek @queue-ref)]
      (alter queue-ref pop)
      item)))

(println "pop: " (qpop q))
(println "pop: " (qpop q))

This gives us the result:

pop:  1
pop:  2

So we have left one value (the integer 3) on the queue.

Next we define a function to watch the queue and run a given s-expression when an item comes on the queue:

Click here to view code image

(defmacro wake-on-queue-event [queue & args]
  (future
   (do
     (loop [x 0]
          (let [item  (qpop (eval queue))]
            (when item
                 (println "pop-wake: " item)
                 (mapcat eval args))); run the instructions passed in when we get
a queue item
          (Thread/sleep 100)
          (recur (inc x))))))

(wake-on-queue-event q (println "awake!"))

The first thing to notice is that everything is wrapped in a future, so it runs off the main thread. Then we have a loop that recurs to infinity, so we need to kill the main thread after a certain point. (You’ll see that we do this later in the program after a duration of 2 seconds). Next we run our qpop function and see if we got a result item. When we have a queue item, we print it saying "pop-wake: ". Then we evaluate our s-expression by mapping over the input args (the s-expression to be run) with the eval function.

We then call this passing in our reference to q and a simple s-expression, with a call to println saying "awake!" This is where we see our blocking code be converted to non-blocking by the macro.

Now there is one item on the queue (the integer 3), so we get the following result:

pop-wake:  3
awake!

Next we trigger our queue watcher by putting some more items on the queue:

(dosync (alter q conj 4 5 6))

This leads to the following result:

pop-wake:  4
awake!
pop-wake:  5
awake!
pop-wake:  6
awake!

Finally, we define our –main function.

Click here to view code image

(defn -main
  "I don't do a whole lot ... yet."
  [& args]
  (future (Thread/sleep 2000) ;put in future-exit after 2 seconds
         (println "Done")
  (System/exit 0))
  (shutdown-agents))

This is called from our Leiningen project file at the following line in the project.clj:

Click here to view code image

  :main ^:skip-aot async-macro.core

This main function lets our program run for 2 seconds and then kills it, saying "Done". We have to kill it because we set our second macro to run to infinity.

The other trick here is our call to shutdown-agents. This is required when using futures to turn off the agent thread pools.

Now in practice you wouldn’t write a macro to do this; you’d make use of the core.async library. For our case, we consider it important to understand how macros do their work as we apply them to solving our blocking code conversion problem.