Chapter 14. Protocols, records, and types
- An overview of the expression problem
- A custom solution to the expression problem
- Clojure’s solution to the expression problem
- A look at deftype, defrecord, reify
Abstraction is an important tenet of software development because it allows code that’s maintainable and extensible. Clojure itself is built on abstractions. For instance, most things in Clojure are coded to interfaces rather than being direct concrete implementations. This allows for reuse of code that expects those interfaces and allows the addition of more implementations to the mix.
Sooner or later, during your time on most projects, you’ll run into an abstraction-oriented issue known as the expression problem. It has to do with how to cleanly extend or use existing code—either something you wrote yourself or, more important, something you don’t own. Clojure’s approach to handling this issue is protocols. In order to get to this topic, though, we’ll first explore the problem in some depth. Then, we’ll come up with our own solution. Finally, we’ll explore protocols, data types, and the reify macro.
14.1. The expression problem
In this section, we’ll explore the expression problem. Instead of going at it from a definitional point of view, we’ll dive into some code. Specifically, we’ll examine a situation that comes up quite often: we need to take two or more classes or sets of classes (or any other abstractions) and make them work together in a seamless way. Further, whatever solution we come up with to do this, there’s usually the need to support further extensibility, perhaps to support even more operations or data types.
Our example code is based on the Clojure programming language and some Java code. It will serve to illustrate the expression problem, as well as lead to ways of solving it.
14.1.1. The Clojure world
Let’s imagine that we’ve been writing an application to manage employee expenses, one that will eventually replace an existing Java application that does a similar job. Naturally, we’re writing it in Clojure. Here’s a part of the expense namespace, with a function to create a simple map containing expense information:
(ns chapter-protocols.expense
(:import [java.text SimpleDateFormat]))
(defn new-expense [date-string dollars cents category merchant-name]
{:date (.parse (SimpleDateFormat. "yyyy-MM-dd") date-string)
:amount-dollars dollars
:amount-cents cents
:category category
:merchant-name merchant-name})
As you’ve seen several times already, using a map is straightforward and is the idiomatic way in Clojure to hold data of any kind. Now, for the purposes of illustration, here’s a function called total-cents that computes the expense amount in cents:
(defn total-cents [e]
(-> (:amount-dollars e)
(* 100)
(+ (:amount-cents e))))
Nothing in this function ought to be unfamiliar, including the threading macro, which you saw in chapter 2.
Let’s also add a function to calculate the total amount, given a list of expenses, and possibly a criteria function by which to select expenses from a list:
(defn total-amount
([expenses-list]
(total-amount (constantly true) expenses-list))
([pred expenses-list]
(->> expenses-list
(filter pred)
(map total-cents)
(apply +))))
Let’s finally add a couple of functions to help us create the predicate functions that can be used with total-amount, specifically to select a particular category of expenses:
(defn is-category? [e some-category] (= (:category e) some-category)) (defn category-is [category] #(is-category? % category))
The second function is syntactic sugar to help create single argument predicates so our code reads easier. Let’s see how it looks by writing some code that tests these functions. Consider a new namespace, such as expense-test, which we’ll start by creating a few sample expenses to play with:
(ns chapter-protocols.expense-test
(:use [chapter-protocols.expense]
[clojure.test]))
(def clj-expenses [(new-expense "2009-8-20" 21 95 "books" "amazon.com")
(new-expense "2009-8-21" 72 43 "food" "mollie-stones")
(new-expense "2009-8-22" 315 71 "car-rental" "avis")
(new-expense "2009-8-23" 15 68 "books" "borders")])
Here’s a test that uses this set of data to compute the total amounts:
(deftest test-clj-expenses-total (is (= 42577 (total-amount clj-expenses))) (is (= 3763 (total-amount (category-is "books") clj-expenses))))
If you run these tests, you’ll see that they pass (if you’d like a refresher on using the clojure.test library or on how to run tests, refer to chapter 8). So now, we have some basic code that shows the intent of our application. You can imagine a lot more functionality that helps an organization track expenses, but we’ve written sufficient code to demonstrate the issues we set out to face, so let’s move on.
14.1.2. The Java world
It’s now time to face a kind of reality we often face when working on application rewrites: business reasons compel us to deal with the old codebase alongside our new one. For the purposes of this example, this means that we’re going to have to deal with instances of the Java-based Expense class. You could imagine such a class being similar to the one shown in the following listing.
Listing 14.1. Skeleton Java class implementing the concept of the expense item
package com.curry.expenses;
import java.util.Calendar;
import java.text.SimpleDateFormat;
import java.text.ParseException;
public class Expense {
private Calendar date;
private int amountDollars;
private int amountCents;
private String merchantName;
private String category;
public Expense(String dateString, int amountDollars, int amountCents,
String category, String merchantName) throws ParseException {
this.date = Calendar.getInstance();
this.date.setTime(new SimpleDateFormat(
"yyyy-MM-dd").parse(dateString));
this.amountDollars = amountDollars;
this.amountCents = amountCents;
this.merchantName = merchantName;
this.category = category;
}
public Calendar getDate() {
return date;
}
public int getAmountDollars() {
return amountDollars;
}
public int getAmountCents() {
return amountCents;
}
public String getMerchantName() {
return merchantName;
}
public String getCategory() {
return category;
}
public int amountInCents() {
return this.amountDollars*100 + this.amountCents;
}
}
To begin working with this class, we’ll write a sanity test to ensure everything is in order. Consider the following:
(def java-expenses [(Expense. "2009-8-24" 44 95 "books" "amazon.com")
(Expense. "2009-8-25" 29 11 "gas" "shell")])
(deftest test-java-expenses-total
(let [total-cents (map #(.amountInCents %) java-expenses)]
(is (= 7406 (apply + total-cents)))))
Again, running this test will result in it passing. Now that we’re able to access the Java class, we need to tackle the situation where we have both kinds of expenses together. For instance, we have to deal with a list of Clojure expense maps constructed via the new-expense function, as well as instances of the com.curry.expenses.Expense class.
We’ll capture this requirement in another test. Consider the following:
(def mixed-expenses (concat clj-expenses java-expenses)) (deftest test-mixed-expenses-total (is (= 49983 (total-amount mixed-expenses))) (is (= 8258 (total-amount (category-is "books") mixed-expenses))))
Now, running these tests won’t pass. Indeed, the first assertion will print a long exception stack trace, because total-amount (and the underlying total-cents and is-category?) function only knows to deal with Clojure map versions of expenses. To fix this, we’re going to have to deal with a design issue.
14.1.3. The expression problem
Philip Wadler is an ACM fellow and a computer science professor at the University of Edinburgh. He has made several important contributions to the field of functional programming, including to the theory behind the Haskell programming language. He also coined the term “expression problem:”:
The Expression Problem is a new name for an old problem. The goal is to define a data-type by cases, where one can add new cases to the data-type and new functions over the data-type, without recompiling existing code, and while retaining static type safety (e.g., no casts).
How can we add functionality to our code (our data type) so it plays well with code (data types) written by someone else (or any other code that we have no control over)? With the previous test we wrote to handle the case of mixed expenses, we’re faced with the expression problem. Specifically, we need the total-amount function to accept and work with an entirely new data type that has its own set of operations (functions) defined for it. We’d also like the category-is function to create functions that can operate on this new data type, even though the new data type has no notion of such a category selector function right now.
The expression problem is common in our industry. There are several approaches to handle the issue, and we’ll briefly look at a few.
Wrappers
Because we have no control over the new data type (the Expense class), this approach creates a new wrapper class around it with the right methods that we can call from our program. The trouble with this approach is that it increases incidental complexity because we’ve added a new class to our system.
First of all, we’ve confused identity: is an object that wraps an instance of Expense identical to it? How should code written elsewhere treat this new wrapper class if it’s passed in? Such an identity crisis is an example of the kind of nonlocal trouble that can arise when an instance of the wrapper is passed to unsuspecting code elsewhere in the system. We’d have to create a wrapper class each time a new data type such as this comes along, leading to an explosion of such wrappers.
When all is said and done, languages such as Java often have no other choice than to go this route.
Monkey patching
Languages such as Ruby are more dynamic and support open classes. This is in contrast with Java’s classes, where once they’ve been written and compiled, they can’t be modified (without manipulating byte code). Open classes, on the other hand, can be changed by anyone using the class, even after the original programmer has moved on. Often, the syntax looks the same as writing the class the first time around, and any new methods defined (or redefined) become part of the original class.
The problem with this approach is that it’s a dangerous one, almost even more so than the wrapper approach. Because all changes to a class happen in a global manner (the class itself being the namespace), it has the potential for collisions. If you open a class and monkey patch it with a new method named total-cents, and someone else comes along and does the same, they’ll overwrite your patch. Such collisions can cause insidious side effects, because they aren’t immediately obvious.
if-then-else
Finally, there’s the approach of not using any well-structured tactic at all and checking for types inline with the code as needed. Client code such as the total-amount function will need to do different things depending on whether it was passed a Clojure map or an instance of the Java Expense class, using good old if-then-else constructs.
This quickly gets complex, depending on how many data types need to be handled. Moreover, if support for a new data type needs to be added at a later point, it isn’t possible without modifying the code in all the places where this type checking is done. The incidental complexity of this approach is too great given that the solution is both rigid and inelegant.
What’s needed is an approach that doesn’t suffer from these problems. The Clojure programming language has the feature for this, and you saw it earlier, in chapter 4. We’re talking about multimethods, and in the next section, we’ll write an implementation that works as desired.
14.1.4. Clojure’s multimethods solution
Multimethods allow us to decouple data types and operations on the data types in an elegant manner. We demonstrated this in chapter 4, where we used multimethods to handle a situation that would’ve required the visitor pattern in a language such as Java. In this section, we’ll use multimethods to get our latest test to pass without modifying the Java code for the Expense class and without creating wrappers or monkey patches.
Let’s refresh our memory by looking at the test that won’t pass right now:
(deftest test-mixed-expenses-total (is (= 49983 (total-amount mixed-expenses))) (is (= 8258 (total-amount (category-is "books") mixed-expenses))))
As we said before, the trouble is that our total-amount, is-category?, and total-cents functions only know how to work with Clojure maps. Our first step, then, will be to address this issue by changing the implementation of the total-cents and is-category? functions. We won’t touch total-amount, because it’s an example of client code (perhaps written by someone using our expense library). We’ll assume that we don’t control it, and indeed, that it’s a requirement of solving the expression problem and we can’t change the alien data type or the client code.
Consider the following code, which is a replacement of the total-cents function:
(defmulti total-cents class)
(defmethod total-cents clojure.lang.IPersistentMap [e]
(-> (:amount-dollars e)
(* 100)
(+ (:amount-cents e))))
Similarly, the following code will serve as the replacement of the is-category? function:
(defmulti is-category? (fn [e category] (class e))) (defmethod is-category? clojure.lang.IPersistentMap [e some-category] (= (:category e) some-category))
We haven’t changed a lot of code; the bodies of the functions are the same as before. All we did was convert the functions to multimethods and redefine the old functions as methods, focusing on the fact that the expense object will be an instance of clojure.lang.IPersistentMap (which all Clojure maps are). Refer to chapter 5 in order to get a refresher on how this works with respect to dispatch functions and dispatch values.
At this point, if we run our tests, the old tests should still pass. Also, the new test will still fail because we haven’t written any code to deal with the Java Expense class. Let’s do that now, starting with the total-cents function:
(defmethod total-cents com.curry.expenses.Expense [e] (.amountInCents e))
And similarly, here’s the is-category? function:
(defmethod is-category? com.curry.expenses.Expense [e some-category] (= (.getCategory e) some-category))
With this, our new test will pass. Note, once again, that we didn’t change the Java Expense class in any way: we didn’t write a wrapper class for it, and we didn’t change the calling code (the total-amount function). We also kept all of our code in our own namespace, allowing others to create their own functions named total-cents and is-category?, without the fear of collisions.
Using multimethods has allowed us to solve this problem of handling new data types in an easy and elegant manner. We’re even set up to deal with more data types now, for example, if we need to ever deal with a third-party expense library.
There are a couple of downsides to this approach, though. The first is that even though multimethods allow us to dispatch via arbitrary functions, we’re using only the class of the first argument, which is either the Clojure map containing expense information or the Java Expense class. We don’t need the full power of multimethods here, and it would be nice if we didn’t have to explicitly write our dispatch functions the way we did previously.
The second issue is that even though the two multimethods we wrote are related to the task of computing totals, it isn’t obvious in the code. If someone were to read this code later, the fact that the two belong together wouldn’t jump out at them. This is even more apparent when you have multiple multimethods that should ideally show some kind of logical grouping. We’ll try to solve these issues next.
14.2. Modus operandi
In this section, we’ll try to solve the two issues we mentioned in the previous section, the first being that we don’t need the conceptual or syntactic complexity of full multi-methods when we only want to dispatch on the class of the first argument. The second is that we want to group related multimethods together so they read better.
We’ll call our solution to this modus operandi, which is a Latin phrase that means “method of operating.” The name reflects our intention here, which is to describe a set of operating procedures for something.
14.2.1. def-modus-operandi
Let’s start with the code we’d like to be able to write:
(def-modus-operandi ExpenseCalculations (total-cents [e]) (is-category? [e category]))
What we’re saying here is that we’re defining a modus operandi called Expense-Calculations that will consist of two methods, namely total-cents and is-category?. We won’t specify the dispatch function as we did before, because we always want it to be the class of the first argument of each method. In this case, both the methods will dispatch based on the class of the expense object, be it a Clojure map or the Java Expense class or any other data type we end up supporting.
Now, let’s look at implementing it. As you can imagine, def-modus-operandi is a macro. Here’s the code along with a couple of associated helper functions:
(defn dispatch-fn-for [method-args]
`(fn ~method-args (class ~(first method-args))))
(defn expand-spec [[method-name method-args]]
`(defmulti ~method-name ~(dispatch-fn-for method-args)))
(defmacro def-modus-operandi [mo-name & specs]
`(do
~@(map expand-spec specs)))
So all we’re doing is generating code that creates multimethods. Here’s what the expanded version looks like:
(do
(clojure.core/defmulti total-cents (clojure.core/fn [e]
(clojure.core/class e)))
(clojure.core/defmulti is-category? (clojure.core/fn [e category]
(clojure.core/class e))))
Notice that the expanded form of is-category? is the same as when we wrote it by hand earlier. The expansion for total-cents is slightly different, only because we can generate the same dispatch function no matter how many arguments the function takes.
Now that we have a way to specify the methods in our modus operandi, we need a way to detail it for the types we’d like to support. We’ll do that next.
14.2.2. detail-modus-operandi
After defining the what of a modus operandi, we need to define the how. We’ll create a new macro called detail-modus-operandi that we’ll use in the following manner:
(detail-modus-operandi ExpenseCalculations
clojure.lang.IPersistentMap
(total-cents [e]
(-> (:amount-dollars e)
(* 100)
(+ (:amount-cents e))))
(is-category? [e some-category]
(= (:category e) some-category)))
Most of the code should be familiar to you, because it’s nearly identical to the code from the previous section. Because all the methods are being defined for the same dispatch value, we’ve made it so that we only have to specify it once. Here’s the implementation of the macro, along with an associated helper function:
(defn expand-method [data-type [name & body]]
`(defmethod ~name ~data-type ~@body))
(defmacro detail-modus-operandi [mo-name data-type & fns]
`(do
~@(map #(expand-method data-type %) fns)))
The expansion of this call to detail-modus-operandi is as follows:
(do
(clojure.core/defmethod total-cents clojure.lang.IPersistentMap [e]
(-> (:amount-dollars e)
(* 100)
(+ (:amount-cents e))))
(clojure.core/defmethod is-category? clojure.lang.IPersistentMap
[e some-category]
(= (:category e) some-category)))
So we’ve done what we set out to do. We have a new abstraction that sits atop multimethods that behave like subtype polymorphism. Our methods dispatch on the type of the first argument.
Notice that even though we specified the name of our modus operandi here (we called it ExpenseCalculations), we haven’t used it for anything. We can make our modus operandi more useful if we use the named objects to track such things as what it contains and who implements it. Let’s do that next.
14.2.3. Tracking our modus operandi
So far, we’ve allowed declarations of a modus operandi that’s a set of related multi-methods that dispatches on the type of the first argument. In this section, we’ll collect some meta information about these methods that we can use to programmatically query things about the modus operandi.
During def-modus-operandi
The first thing we’ll do is to define a var with the name of the modus operandi. Doing that by itself is easy enough: we add a call to def in our def-modus-operandi macro. The question is, what should the var be bound to? A simple option is to create a map containing information about the modus operandi. Let’s try that approach:
(defmacro def-modus-operandi [mo-name & specs]
`(do
(def ~mo-name ~(mo-methods-registration specs))
~@(map expand-spec specs)))
We’ve delegated to a helper function called mo-methods-registration, so let’s implement that next:
(defn mo-method-info [[name args]]
{(keyword name) {:args `(quote ~args)}})
(defn mo-methods-registration [specs]
(apply merge (map mo-method-info specs)))
We’re collecting the name and arguments of each method into a map. This map, with all the information about the methods being specified as part of the modus operandi, will become the root binding of a var by the same name as the modus operandi. Let’s try it. First, we’ll redefine the modus operandi:
user> (def-modus-operandi ExpenseCalculations
(total-cents [e])
(is-category? [e category]))
#'user/is-category?
Next, let’s see what the ExpenseCalculation var is bound to:
user> ExpenseCalculations
{:is-category? {:args [e category]}, :total-cents {:args [e]}}
So we have the basic information. Next, we’ll collect some more information every time detail-modus-operandi is called.
During detail-modus-operandi
To collect information of the implementer of a modus operandi, we’ll first need to pass the modus operandi into the expand-method function:
(defmacro detail-modus-operandi [mo-name data-type & fns]
`(do
~@(map #(expand-method mo-name data-type %) fns)))
Now that our expand-method knows what modus operandi it’s going to create a method for, we can collect information about it:
(defn expand-method [mo-name data-type [method-name & body]]
`(do
(alter-var-root (var ~mo-name) update-in
[(keyword '~method-name) :implementors] conj ~data-type)
(defmethod ~method-name ~data-type ~@body)))
To better understand this addition to the expand-method function, let’s talk about the data we’re collecting. Recall that the modus operandi var is bound to a map that contains a key for each method. The value for each such key is another map. The only key in the inner map so far is :args, and to collect the data types of the implementors to this map, we’ll introduce another key called :implementors. So here we’re going to conj the data type onto the list of implementors (if any) each time a method of a modus operandi is implemented. Finally, let’s look at the function alter-var-root. Here’s the doc string:
user> (doc alter-var-root)
clojure.core/alter-var-root
([v f & args])
Atomically alters the root binding of var v by applying f to its current
value plus any args
So we’re passing it the var for the modus operandi and the function update-in. The argument to update-in is a sequence of keys that locates a nested value and a function that will be applied to the existing value along with any other arguments. In this case, update-in is passed the function conj along with the data type we’d like recorded.
Phew, that’s a lot of work for a single line of code. The following listing shows the complete implementation of modus operandi in a single namespace.
Listing 14.2. Implementing modus operandi on top of multimethods
(ns chapter-protocols.modus-operandi)
(defn dispatch-fn-for [method-args]
`(fn ~method-args (class ~(first method-args))))
(defn expand-spec [[method-name method-args]]
`(defmulti ~method-name ~(dispatch-fn-for method-args)))
(defn mo-method-info [[name args]]
{(keyword name) {:args `(quote ~args)}})
(defn mo-methods-registration [specs]
(apply merge (map mo-method-info specs)))
(defmacro def-modus-operandi [mo-name & specs]
`(do
(def ~mo-name ~(mo-methods-registration specs))
~@(map expand-spec specs)))
(defn expand-method [mo-name data-type [method-name & body]]
`(do
(alter-var-root (var ~mo-name) update-in [(keyword '~method-name)
:implementors] conj ~data-type)
(defmethod ~method-name ~data-type ~@body)))
(defmacro detail-modus-operandi [mo-name data-type & fns]
`(do
~@(map #(expand-method mo-name data-type %) fns)))
Let’s look at it in action. First, let’s make a call to detail-modus-operandi:
user> (detail-modus-operandi ExpenseCalculations
clojure.lang.IPersistentMap
(total-cents [e]
(-> (:amount-dollars e)
(* 100)
(+ (:amount-cents e))))
(is-category? [e some-category]
(= (:category e) some-category)))
#<MultiFn clojure.lang.MultiFn@4aad8dbc>
Now let’s look at our ExpenseCalculations var:
user> ExpenseCalculations
{:is-category? {:implementors (clojure.lang.IPersistentMap),
:args [e category]},
:total-cents {:implementors (clojure.lang.IPersistentMap),
:args [e]}}
As you can see, we’ve added the new :implementors key to the inner maps, and they have a value that’s a sequence of the implementors so far. Let’s implement the modus operandi for the Java Expense class now:
user> (detail-modus-operandi ExpenseCalculations
com.curry.expenses.Expense
(total-cents [e]
(.amountInCents e))
(is-category? [e some-category]
(= (.getCategory e) some-category)))
#<MultiFn clojure.lang.MultiFn@4aad8dbc>
Let’s now see what our ExpenseCalculations var is bound to:
user> ExpenseCalculations
{:is-category? {:implementors (com.curry.expenses.Expense
clojure.lang.IPersistentMap),
:args [e category]},
:total-cents {:implementors (com.curry.expenses.Expense
clojure.lang.IPersistentMap),
:args [e]}}
And there you have it: we’re collecting a sequence of all implementing classes inside the map bound to the modus-operandi var. Let’s now ensure that everything still works with our original code. Figure 14.1 shows a conceptual view of the process of defining a modus operandi and then detailing it out. Listing 14.3 shows the complete code for our expense namespace.
Figure 14.1. Calling def-modus-operandi creates a var that will hold information about the modus operandi, which can later be used to introspect it. The macro itself makes as many calls to defmulti as needed. The detail-modus-operandi macro is the other side of the modus operandi concept: it fills out the implementation details by expanding to as many defmethod calls as specified. It also updates the modus-operandi var to reflect the implementor information.
Listing 14.3. The expense namespace using the modus operandi multimethod syntax
(ns chapter-protocols.expense-modus-operandi
(:use chapter-protocols.modus-operandi)
(:import [java.text SimpleDateFormat]
[java.util Calendar]))
(defn new-expense [date-string dollars cents category merchant-name]
(let [calendar-date (Calendar/getInstance)]
(.setTime calendar-date
(.parse (SimpleDateFormat. "yyyy-MM-dd") date-string))
{:date calendar-date
:amount-dollars dollars
:amount-cents cents
:category category
:merchant-name merchant-name}))
(def-modus-operandi ExpenseCalculations
(total-cents [e])
(is-category? [e category]))
(detail-modus-operandi ExpenseCalculations
clojure.lang.IPersistentMap
(total-cents [e]
(-> (:amount-dollars e)
(* 100)
(+ (:amount-cents e))))
(is-category? [e some-category]
(= (:category e) some-category)))
(detail-modus-operandi ExpenseCalculations
com.curry.expenses.Expense
(total-cents [e]
(.amountInCents e))
(is-category? [e some-category]
(= (.getCategory e) some-category)))
(defn category-is [category]
#(is-category? % category))
(defn total-amount
([expenses-list]
(total-amount (constantly true) expenses-list))
([pred expenses-list]
(->> expenses-list
(filter pred)
(map total-cents)
(apply +))))
Similarly, the following listing shows the tests we wrote so far, all in one place. We’ll run the tests next.
Listing 14.4. Testing the implementation of modus operandi calculating expense totals
(ns chapter-protocols.expense-test
(:import [com.curry.expenses Expense])
(:use [chapter-protocols.expense-modus-operandi]
[clojure.test]))
(def clj-expenses [(new-expense "2009-8-20" 21 95 "books" "amazon.com")
(new-expense "2009-8-21" 72 43 "food" "mollie-stones")
(new-expense "2009-8-22" 315 71 "car-rental" "avis")
(new-expense "2009-8-23" 15 68 "books" "borders")])
(deftest test-clj-expenses-total
(is (= 42577 (total-amount clj-expenses)))
(is (= 3763 (total-amount (category-is "books") clj-expenses))))
(def java-expenses [(Expense. "2009-8-24" 44 95 "books" "amazon.com")
(Expense. "2009-8-25" 29 11 "gas" "shell")])
(deftest test-java-expenses-total
(let [total-cents (map #(.amountInCents %) java-expenses)]
(is (= 7406 (apply + total-cents)))))
(def mixed-expenses (concat clj-expenses java-expenses))
(deftest test-mixed-expenses-total
(is (= 49983 (total-amount mixed-expenses)))
(is (= 8258 (total-amount (category-is "books") mixed-expenses))))
user> (use 'clojure.test) (run-tests 'chapter-protocols.expense-test)
Testing chapter-protocols.expense-test
Ran 3 tests containing 5 assertions.
0 failures, 0 errors.
{:type :summary, :test 3, :pass 5, :fail 0, :error 0}
Finally, before wrapping up this section, let’s write a couple of functions that will make it easy to query data about our modus operandi, like ExpenseCalculations.
Querying modus operandi
The first function we’ll write discerns what data types implement a particular modus operandi. Consider this code:
(defn implementors [modus-operandi method] (get-in modus-operandi [method :implementors]))
And this allows us to do things like this:
user> (implementors ExpenseCalculations :is-category?) (com.curry.expenses.Expense clojure.lang.IPersistentMap)
Let’s write another function that when given a class of a particular data type can tell us if it implements a particular method of a modus operandi. Here’s the code:
(defn implements? [implementor modus-operandi method]
(some #{implementor} (implementors modus-operandi method)))
Let’s test it at the REPL:
user> (implements? com.curry.expenses.Expense ExpenseCalculations
:is-category?)
com.curry.expenses.Expense
Note that implements? returns the class itself, which is truthy. Here’s a negative scenario:
user> (implements? java.util.Date ExpenseCalculations :is-category?) nil
Now that we have a function such as implements?, we can also write a broader function to see if a class implements a modus operandi completely:
(defn full-implementor? [implementor modus-operandi]
(->> (keys modus-operandi)
(map #(implements? implementor modus-operandi %))
(not-any? nil?)))
Here it is in action:
user> (full-implementor? com.curry.expenses.Expense ExpenseCalculations) true
To test the negative side, let’s partially implement the modus operandi:
user> (detail-modus-operandi ExpenseCalculations
java.util.Date
(total-cents [e]
(rand-int 1000)))
#<MultiFn clojure.lang.MultiFn@746ac18c>
And now we can test what we were after:
user> (full-implementor? java.util.Date ExpenseCalculations) false
We can implement other functions such as these, because the value bound to the modus-operandi var is a regular map that can be inspected like any other. We’re nearly at the end of this section. Before moving on though, let’s examine the downsides to our modus operandi approach to the expression problem.
14.2.4. The next step
In this section, we took multimethods and wrote a little DSL on top of them that allows us to write simpler, clearer code when we want to dispatch on the class of the first argument. We were also able to group related multimethods together via this new syntax, and this allowed the code to be self-documenting by communicating that certain multimethods are related to each other.
What we haven’t touched on at all is error handling. For instance, if you eval the same detail-modus-operandi calls multiple times, our data-collection functions would add the class to our modus operandi metadata map multiple times. It’s an easy fix, but this isn’t the most robust code in the world, because we wrote it to demonstrate the abstraction.
There are other trouble spots as well. For instance, because we built this on top of multimethods, and multimethods support hierarchies (and Java inheritance hierarchies by default), our implements? and related functions won’t give accurate answers as they stand now.
Further, because this is such a barebones implementation, many other features might be desirable in a more production-ready version. The other downside is that there’s a small performance hit when using multimethods because they have to call the dispatch function and then match the dispatch value against available multimethods. After all, our approach is syntactic sugar on top of multimethods.
In the next section, you’ll see Clojure’s version of the solution.
14.3. Protocols and data types
You’ve already seen what the expression problem is and a variety of ways to solve it. Clojure’s multimethods are perfectly suited to writing code that allows independent extension of the supported data types and operations. We also created an abstraction called modus operandi that supports the most common use of multimethods, that of single dispatch (the first argument) based on the type (or class).
Because Clojure is written in Java, and the multimethod mechanism is implemented in Clojure itself, there’s a performance penalty to be paid every time a multi-method is called. In most cases, this is negligible, and the increased expressiveness of code more than makes up for it. But as Clojure matures and moves more of its implementation into Clojure itself, there needs to be a way to support its abstraction and data-definition facilities without this performance hit. Protocols and data types are that solution, and they also offer a high-performance solution to the commonly encountered version of the expression problem.
In this section, we’ll examine what protocols and data types are and how they can be used. Keep in mind our design of modus operandi as we work through this.
14.3.1. defprotocol and extend-protocol
The word protocol means the way something is done, often predefined and followed by all participating parties. Clojure protocols are analogous to our modi operandi, and defprotocol is to protocols what def-modus-operandi is to modi operandi. Similarly, extend-protocol is to protocols what detail-modus-operandi is to modi operandi. Listing 14.3 showed the implementation of the expense calculation, and the next listing shows the same logic implemented using Clojure protocols.
Listing 14.5. The expense namespace using a Clojure protocol
(ns chapter-protocols.expense-protocol
(:import [java.text SimpleDateFormat]
[java.util Calendar]))
(defn new-expense [date-string dollars cents category merchant-name]
(let [calendar-date (Calendar/getInstance)]
(.setTime calendar-date (.parse (SimpleDateFormat. "yyyy-MM-dd")
date-string))
{:date calendar-date
:amount-dollars dollars
:amount-cents cents
:category category
:merchant-name merchant-name}))
(defprotocol ExpenseCalculations
(total-cents [e])
(is-category? [e category]))
(extend-protocol ExpenseCalculations
clojure.lang.IPersistentMap
(total-cents [e]
(-> (:amount-dollars e)
(* 100)
(+ (:amount-cents e))))
(is-category? [e some-category]
(= (:category e) some-category)))
(extend-protocol ExpenseCalculations
com.curry.expenses.Expense
(total-cents [e]
(.amountInCents e))
(is-category? [e some-category]
(= (.getCategory e) some-category)))
(defn category-is [category]
#(is-category? % category))
(defn total-amount
([expenses-list]
(total-amount (constantly true) expenses-list))
([pred expenses-list]
(->> expenses-list
(filter pred)
(map total-cents)
(apply +))))
The only thing that’s different from the implementation based on modus operandi is that we’ve removed the dependence on the chapter-protocols.modus-operandi namespace, and we’ve replaced the calls to def-modus-operandi and detail-modus-operandi with calls to defprotocol and extend-protocol. At a conceptual level, the code in listing 14.5 should make sense. We’ll get into the specifics now.
Defining new protocols
As you might have guessed, new protocols are defined using the defprotocol macro. It defines a set of named methods, along with their signatures. Here’s the official syntax:
(defprotocol AProtocolName
;optional doc string
"A doc string for AProtocol abstraction"
;method signatures
(bar [this a b] "bar docs")
(baz [this a] [this a b] [this a b c] "baz docs"))
The protocol as well as the methods that form it can accept doc strings. A call to defprotocol results in a bunch of vars being created: one for the protocol itself and one for each polymorphic function (or method) that’s a part of the protocol. These functions dispatch on the type of the first argument (and therefore must have at least one argument), and by convention, the first argument is called this. So from listing 14.5, the following snippet defines a protocol named ExpenseCalculations:
(defprotocol ExpenseCalculations (total-cents [e]) (is-category? [e category]))
We’re defining a set of related methods (total-cents and is-category?) that can be implemented any number of times by any data type. A call to defprotocol also generates an underlying Java interface. So, because the previous code exists in the namespace chapter-protocols.expense-protocol, it will result in a Java interface called chapter_protocols.expense_protocol.ExpenseCalculations. The methods in this interface will be the ones specified in the definition of the protocol, total_cents and is_category_QMARK_, the latter of which is the translated name of a Clojure function (one that ends with a question mark) into Java. The fact that defprotocol generates a Java interface also means that if some other Java code wants to participate in a protocol, it can implement the generated interface and proceed as usual.
Now that we’ve defined a protocol, any data type can participate in it.
Participating in protocols
Having defined a protocol, let’s see how you can use it. As an example, consider the call to extend-protocol, also from listing 14.5:
(extend-protocol ExpenseCalculations
com.curry.expenses.Expense
(total-cents [e]
(.amountInCents e))
(is-category? [e some-category]
(= (.getCategory e) some-category)))
This means that the com.curry.expenses.Expense data type will participate in the ExpenseCalculations protocol, and when either total-cents or is-category? is called with an instance of this class as the first argument, it will be correctly dispatched to the previous implementation.
You can also specify more than one participant at a time; you can define the implementations of the protocol methods for more than a single data type. Here’s an example:
(extend-protocol ExpenseCalculations
clojure.lang.IPersistentMap
(total-cents [e]
(-> (:amount-dollars e)
(* 100)
(+ (:amount-cents e))))
(is-category? [e some-category]
(= (:category e) some-category))
com.curry.expenses.Expense
(total-cents [e]
(.amountInCents e))
(is-category? [e some-category]
(= (.getCategory e) some-category)))
We’ll now look at another way to specify how data types can participate in protocols.
The extend-type macro
extend-protocol is a helper macro, defined on top of another convenient macro named extend-type. It’s sort of the other way of specifying a participant of a protocol, in that it focuses on the data type. Here’s an example of extend-type in use:
(extend-type com.curry.expenses.Expense
ExpenseCalculations
(total-cents [e]
(.amountInCents e))
(is-category? [e some-category]
(= (.getCategory e) some-category)))
Again, because a single data type can participate in multiple protocols, extend-type lets you specify any number of protocols. Although extend-protocol and extend-type make it quite easy to use protocols, they both ultimately resolve to calls to the extend function.
The extend function
The extend function lives in Clojure’s core namespace, and it’s the one that does the work of registering protocol participants and associating the methods with the right data types. Here’s an example of the extend function in action:
(extend com.curry.expenses.Expense
ExpenseCalculations {
:total-cents (fn [e]
(.amountInCents e))
:is-category? (fn [e some-category]
(= (.getCategory e) some-category))})
This might look similar to the code we generated in our implementation of modus operandi. For each protocol and data type pair, extend accepts a map that describes participation of that data type in the protocol. The keys of the map are keyword versions of the names of the methods, and the values are the function bodies that contain the implementation for each. The extend function is the most flexible in terms of building an implementation of a protocol.
Figure 14.2 shows the conceptual flow of defining and using protocols, in a manner analogous to figure 14.1.
Figure 14.2. Calling defprotocol performs an analogous operation where a var is created to hold information about the protocol and its implementors. The underlying implementation will also result in a Java interface that pertains to the protocol being defined. Calls to extend, extend-type, and extend-protocol will update the var with implementor details and generate Java classes that implement the protocol.
We’ve covered protocols and how they’re defined and used. We’ll say another couple of things about them before moving on to the remaining topics of this chapter.
Protocols and nil
You’ve seen that protocol methods are dispatched based on the class of the first argument. A natural question arises: what will happen if the first argument to a protocol method is nil? What is the class of nil?
user> (class nil) nil
If you call a protocol method, say total-cents from our expense example with nil, you’ll get an error complaining that no implementation was found. Luckily, protocols can be extended on nil:
(extend-protocol ExpenseCalculations nil (total-cents [e] 0))
After this, calling total-cents with nil will return zero. Our last stop in this section will be to explore a few functions that help us reflect on defined protocols.
Reflecting on protocols
Sometimes it’s useful to programmatically reflect on specific protocols and their extenders. When we wrote modus operandi, we also wrote some helper functions that let us reflect on implements?, implementors, and full-implementor. Clojure protocols also have functions that work in a similar fashion. Let’s take a look at them.
user> (extends? ExpenseCalculations com.curry.expenses.Expense) true user> (extends? ExpenseCalculations clojure.lang.IPersistentMap) true user> (extends? ExpenseCalculations java.util.Date) false
Needless to say, the function extends? can be used to check if a particular data type participates in a given protocol. The next function that’s useful around such querying is extenders:
user> (extenders ExpenseCalculations) (nil com.curry.expenses.Expense clojure.lang.IPersistentMap)
Again, the extenders function lists all the data types that participate in a particular protocol. The final function of interest is called satisfies? and it works like this:
user> (satisfies? ExpenseCalculations (com.curry.expenses.Expense. "10-10-
2010" 20 95 "books" "amzn"))
true
user> (satisfies? ExpenseCalculations (new-expense "10-10-2010" 20 95 "books"
"amzn"))
true
user> (satisfies? ExpenseCalculations (java.util.Random.))
false
Note that the satisfies? function works on instances of extenders, not extender data types themselves. You’ll find this function even more useful once you’ve seen the reify macro in action, which we’ll explore in the next section. We’ve now covered all the topics about protocols that we set out to cover. The next section is about the other side of this picture; we’ll review a couple of ways to define data types.
14.3.2. deftype, defrecord, and reify
We started this chapter by considering the expression problem, and as you might recall, there are two sides to it, namely, data types and the operations on them. So far, we’ve been looking primarily at the operations side of the situation; in this section we’ll look at a couple of ways to define data types.
The mechanisms we’re going to talk about create underlying classes on the host platform (namely Java today, but it could be others tomorrow). This means that they share the same performance as the native version of such data types, as well as the same polymorphic capabilities supported by the host. We’ll first look at defrecord, followed by deftype. We’ll close the section with a look at reify.
Defrecord
Let’s start with an example of using defrecord:
(defrecord NewExpense [date amount-dollars amount-cents
category merchant-name])
As we mentioned, this call defines a named class (chapter-protocols.expense-record.NewExpense) that has the specified set of fields with names as specified in the call to defrecord. Because this is a proper class on the host environment, the type of class is fully specified and known, allowing for a high-performance dispatch of fields and methods. Similarly, it has a named constructor, similar to other Java classes. Here’s how you’d create an instance of the NewExpense data type:
user> (import chapter-protocols.expense-record.NewExpense)
chapter-protocols.expense-record.NewExpense
user> (NewExpense. "2010-04-01" 29 95 "gift" "1-800-flowers")
#:chapter-protocols.expense-record.NewExpense{:date "2010-04-01",
:amount-dollars 29,
:amount-cents 95,
:category "gift",
:merchant-name "1-800-flowers"}
Notice that the printer shows this object as being an instance of the generated class, as expected. The only downside to this is that the reader can’t reconstruct this if asked to read a string containing this text. This support may be added in a future version.
Now that we’ve come this far, let’s go ahead and change the implementation of our expense namespace to use this. The following listing shows the new implementation.
Listing 14.6. The expense namespace using a Clojure protocol and defrecord
(ns chapter-protocols.expense-record
(:import [java.text SimpleDateFormat]
[java.util Calendar]))
(defrecord NewExpense [date amount-dollars amount-cents
category merchant-name])
(defn new-expense [date-string dollars cents category merchant-name]
(let [calendar-date (Calendar/getInstance)]
(.setTime calendar-date (.parse (SimpleDateFormat. "yyyy-MM-dd")
date-string))
(NewExpense. calendar-date dollars cents category merchant-name)))
(defprotocol ExpenseCalculations
(total-cents [e])
(is-category? [e category]))
(extend-type NewExpense
ExpenseCalculations
(total-cents [e]
(-> (:amount-dollars e)
(* 100)
(+ (:amount-cents e))))
(is-category? [e some-category]
(= (:category e) some-category)))
(extend com.curry.expenses.Expense
ExpenseCalculations {
:total-cents (fn [e] (.amountInCents e))
:is-category? (fn [e some-category] (= (.getCategory e)
some-category))})
(extend-protocol ExpenseCalculations nil
(total-cents [e] 0))
(defn category-is [category]
#(is-category? % category))
(defn total-amount
([expenses-list]
(total-amount (constantly true) expenses-list))
([pred expenses-list]
(->> expenses-list
(filter pred)
(map total-cents)
(apply +))))
Notice the call to extend-type and how we use the name of our newly defined record NewExpense instead of the previously used, more generic IPersistentMap. This shows that records can participate fully in protocols and indeed can participate in as many as needed. By the way, for completeness, it’s worth modifying the test namespace to depend on the new chapter-protocols.expense-record namespace and checking to see if all tests pass. They should.
Notice how you can access the fields of the NewExpense instances using keywords. This is because defrecord creates a class that already implements several interfaces, including IPersistentMap, IKeywordLookup, and ILookup. In this manner, they work in the same way as regular Clojure maps, including with respect to destructuring, metadata, and the use of functions such as assoc and dissoc. A useful point to note is that records are extensible in that they can accept values for keys that weren’t originally specified as part of the defrecord call. The only penalty to this is that such keys have the same performance as Clojure maps. Records also implement the hashCode and equals methods, in order to support value-based equality out of the box. A final note is that the field specification supports type hints.
In listing 14.6, you’ve already seen how records can participate in protocols. We made nearly no change to our code from the previous implementation from listing 14.5, but records have more direct support for protocols. They can supply the implementations of protocols inline with their definition. The following listing shows this version.
Listing 14.7. The expense namespace with defrecord and inline protocol
(ns chapter-protocols.expense-record-2
(:import [java.text SimpleDateFormat]
[java.util Calendar]))
(defprotocol ExpenseCalculations
(total-cents [e])
(is-category? [e category]))
(defrecord NewExpense [date amount-dollars amount-cents
category merchant-name]
ExpenseCalculations
(total-cents [this]
(-> amount-dollars
(* 100)
(+ amount-cents)))
(is-category? [this some-category]
(= category some-category)))
(defn new-expense [date-string dollars cents category merchant-name]
(let [calendar-date (Calendar/getInstance)]
(.setTime calendar-date (.parse (SimpleDateFormat. "yyyy-MM-dd")
date-string))
(NewExpense. calendar-date dollars cents category merchant-name)))
(extend com.curry.expenses.Expense
ExpenseCalculations {
:total-cents (fn [e] (.amountInCents e))
:is-category? (fn [e some-category] (= (.getCategory e)
some-category))})
(extend-protocol ExpenseCalculations nil
(total-cents [e] 0))
(defn category-is [category]
#(is-category? % category))
(defn total-amount
([expenses-list]
(total-amount (constantly true) expenses-list))
([pred expenses-list]
(->> expenses-list
(filter pred)
(map total-cents)
(apply +))))
The main change is in the following snippet:
(defrecord NewExpense [date amount-dollars amount-cents
category merchant-name]
ExpenseCalculations
(total-cents [this]
(-> amount-dollars
(* 100)
(+ amount-cents)))
(is-category? [this some-category]
(= category some-category)))
Notice that we followed the field names with the protocol name that we wish to implement. The protocol name is followed by the implementations of the protocol methods. You can similarly follow that with more protocol specifications (the protocol name followed by the implementation).
Java support
What’s more, this isn’t restricted to protocols; you can also specify and implement Java interfaces. The code would look similar to the previous protocol specification: you’d specify the interface name followed by the implementation of the interface methods. Instead of a protocol or an interface, you can also specify object and override methods from the Object class. Recall that the first parameter of all protocol methods is the implementor instance itself, so you must pass the conventionally named this parameter as before. This means that there will be one more parameter for each interface or object method when compared to the corresponding definition. Finally, if the method implementation needs to call recur, the this parameter shouldn’t be passed, because it will be passed automatically.
With this discussion, we’ve covered records. They can be used in all places where maps might have been used because they’re faster and also support protocols. Note that our implementation of protocol methods didn’t result in closures, and when this functionality is needed, you can use the reify macro. You’ll see that shortly, but our next stop is the deftype macro.
Deftype
When you use defrecord, you get a whole bunch of functionality for free. You get map-like behavior of using keywords to look stuff up, you get value-based equality behavior, you get metadata support, and you get serialization. This is usually exactly what you need when developing application domain data types, such as our expense data type from the previous few sections.
But there are times when you don’t need any of this; indeed, at times you want to specify your own implementations for some of these interfaces. It’s for these times that Clojure also provides the deftype macro:
(deftype Mytype [a b])
This generates an underlying Java class that looks like this:
public final class Mytype {
public final Object a;
public final Object b;
public Mytype(Object obj, Object obj1) {
a = obj;
b = obj1;
}
}
As you can see, the fundamental difference between defrecord and deftype is that the latter produces a bare-metal class that you can do whatever you want with. The most common use case of deftype is to build infrastructure abstractions. Examples of such an abstraction might be a special collection to hold your domain-specific objects or a custom transaction manager. When you do need such a data type, with the performance characteristics of the native host, you can use deftype. In most other cases, defrecord should suffice.
We’re nearly finished! In the previous section, we briefly mentioned closures. In the next section, we’ll show how to create anonymous data types and instances of them using the reify macro.
Reify
Reification means to bring something into being or to turn something into a concrete form. The reify macro takes a protocol, which by itself is an abstract set of methods, and creates a concrete instance of an anonymous data type that implements that protocol. It does so with the full power of Clojure’s lexical closures. For example, you might implement the new-expense function as follows:
(defn new-expense [date-string dollars cents category merchant-name]
(let [calendar-date (Calendar/getInstance)]
(.setTime calendar-date
(.parse (SimpleDateFormat. "yyyy-MM-dd") date-string))
(reify ExpenseCalculations
(total-cents [this]
(-> dollars
(* 100)
(+ cents)))
(is-category? [this some-category]
(= category some-category)))))
In a pattern that’s similar to one you’ve seen before, reify accepts one or more protocols and their implementation. In this example, reify was passed the ExpenseCalculations protocol along with the implementations of the total-cents and is-category? methods. The object returned by reify is a closure; in the case of new-expense, the lexically bound closure includes the parameters passed to new-expense, along with the names created in the let form.
Here’s another example from the chapter on messaging using RabbitMQ. The following snippet is taken from listing 11.4:
(defn on-swarm [worker-name args]
(let [worker-data (ref worker-init-value)
worker-transport (dispatch-work worker-name args worker-data)]
(fn [accessor]
(condp = accessor
:complete? (not (= worker-init-value @worker-data))
:value (attribute-from-response @worker-data :value)
:status (@worker-data :status)
:disconnect (disconnect-worker worker-transport)))))
This on-swarm function returns a closure that accepts command keywords such as :value and :status. An example of using this closure is the all-complete? function, also taken from the same listing:
(defn all-complete? [swarm-requests] (every? #(% :complete?) swarm-requests))
The parameter swarm-requests is a sequence of closures returned by the on-swarm function. We’ll rewrite this using a protocol. First, we’ll define one:
(defprotocol RemoteWorker (complete? [rw]) (value [rw]) (status [rw]) (disconnect [rw]))
Next, we’ll redefine on-swarm:
(defn on-swarm [worker-name args]
(let [worker-data (ref worker-init-value)
worker-transport (dispatch-work worker-name args worker-data)]
(reify RemoteWorker
(complete? [rw]
(not (= worker-init-value @worker-data)))
(value [rw]
(attribute-from-response @worker-data :value))
(status [rw]
(@worker-data :status))
(disconnect [rw]
(disconnect-worker worker-transport)))))
We’ll also fix places where the closure was being used in the old way, such as the all-complete? function:
(defn all-complete? [swarm-requests] (every? complete? swarm-requests))
This is a far more natural way to write this function. The reify macro is useful whenever you have a situation such as the one here, where it makes sense to create and return a closure.
The benefit of using protocols here is that you can create new versions of objects that implement this protocol, such as one that doesn’t delegate the computation to a remote worker process but executes it locally (useful for testing) or one that uses an entirely different transport mechanism (instead of RabbitMQ). As far as the client code is concerned, it will keep working as long as an extender of RemoteWorker is passed in.
You’ve now learned enough about protocols and data types to use them in your own programs. In order to round off this chapter, we’ll make a few observations about protocols and compare them to multimethods.
14.4. Summary
Protocols were originally introduced to satisfy the need for low-level implementation techniques that would be fast enough to implement the language itself in, a la Clojure in Clojure. They also serve to solve 90% of the expression problem cases, where class-based single dispatch is acceptable. In this way, they’re less powerful than multimethods.
Even with that, protocols have several advantages. Similar to multimethods, they don’t tie polymorphism to inheritance. They allow grouping of related methods into a conceptual unit, which makes for clearer, self-documenting code. Because protocols generate interfaces from the underlying host, they’re able to provide performance that’s on par with the host itself. Similar to multimethods, they’re an open way of solving the expression problem. This means that new data types and operations can be added while making minimum changes to existing code. Similarly, openness is maintained with respect to who is allowed to participate in protocols. Any number of data types can implement a single protocol, and a data type can implement any number of protocols. Finally, because protocols belong to the namespace they’re defined in, there’s no danger of name collisions if someone defines a protocol with the same name that you chose.
As a parting note, it’s worth mentioning that even before the introduction of defrecord, using maps to store information was the idiomatic way to implement things. Hiding information behind non-generic interfaces (such as getters/setters or even more custom methods) makes such information less reusable by code that wasn’t designed to access such an API. Maps provide far more generic manipulability, and records take that one step further by making it perform as fast as the host platform can make it.
When coupled with protocols, your Clojure code will be built on abstractions. This will ensure that it’s more flexible and easier to maintain, as well as being easy for other people to work with. In this manner, protocols give a huge benefit even beyond solving the most common case of the expression problem.