Chapter 41. Dynamic Reception

Handle messages without defining them in the receiving class.

Also known as: Overriding method_missing or doesNotUnderstand

Any object has a limited set of methods defined for it. A client of an object may attempt to invoke a method that isn’t defined on the receiver. A statically typed language will spot this at compile time and report a compilation error. As a result, you know you won’t get this kind of error at runtime (unless you do some clever fiddling to get around the type system). With a dynamically typed language, you can invoke a nonexistent method at runtime, which usually gives you a runtime error.

Dynamic Reception allows you to adjust this behavior, which means you can respond differently to an unknown message.

41.1 How It Works

Many dynamic languages react to an unknown method invocation by calling a special error-handling method at the top of the object hierarchy. There is no standard name for this method: in Smalltalk it’s doesNotUnderstand, and in Ruby it’s method_missing. You can introduce your own processing for an unknown method by subclassing these methods in your own class. When you do this, you are essentially dynamically altering the rules for the reception of method calls.

There are many reasons why Dynamic Reception is useful in general programming. One excellent example is supporting automatic delegation to another object. To do this, you define the methods that you wish to handle in the original receiver and use Dynamic Reception to send any unknown messages to a delegate object.

Dynamic Reception can feature in several ways in DSL work. One common use is to convert what might otherwise be method parameters into the name of the method. A good example of this is Rails’s Active Record dynamic finders. Say you have a Person class with firstname and lastname properties. With these defined, you can call find_by_firstname("martin") or find_by_firstname_and_lastname("martin", "fowler") without having to define these methods. The code in the Active Record superclass overrides Ruby’s method_missing and checks to see if the method call begins with find_by. If so, it then parses the method name to find the property names and uses them to construct a query. You could also do this by passing in multiple arguments, such as find_by("firstname", "martin", "lastname", "fowler"), but putting the property names into the method name is more readable as it mimics what you would actually do if you explicitly defined methods like that.

A method like find_by_name works by taking a single method name and parsing it. Essentially, you are embedding an external DSL in the method name. Another approach is to use a sequence of Dynamic Receptions, like find_by.firstname("martin").and.lastname("fowler") or find_by.firstname.martin.and.lastname. fowler. In this case, the find_by method would return an Expression Builder that you can use to build up a query using Method Chaining and Dynamic Reception.

One of the advantages of doing this is that it can avoid quoting the various parameters, using martin rather than "martin" to reduce noise. If you are using Object Scoping, with this mechanism you can use bare symbols for arguments, for example state idle instead of state :idle. You can do this by implementing Dynamic Reception in the superclass so that, once the object has had state invoked, it will override the next unknown method call to capture the name of the state. You can go further by using Textual Polishing to remove various bits of noisy punctuation.

41.2 When to Use It

Using Dynamic Reception to move parameters into method names is appealing for a couple of reasons. First, it can mimic what you would genuinely do with a method but with less effort. It’s quite reasonable to imagine a person class having a find_by_firstname_and_lastname method; by using Dynamic Reception you are providing this method without having to actually program it. This can be a significant time-saver, particularly if you are using lots of combinations. Certainly, there are other ways you can do it; you can put attribute name into the parameters as in find(:firstname, "martin", :lastname, "fowler"), use a closure like find {|p| p.firstname == "martin" and p.lastname == "fowler"}, or even embed a fragmentary external DSL in a string as in find("firstname == martin lastname == fowler"). Yet many people find that embedding the field names into the method names is the most fluent way to express the call.

Another benefit of replacing parameters by method names is that it can give you better consistency in punctuation. An expression like find.by.firstname. martin.and.lastname.fowler uses dots as the only form of punctuation. This is good, in that people won’t get confused as to when they should use a dot and when a pair of parentheses or quotes. For many others, this consistency isn’t a virtue; I like separating what is schema from what is data, so I prefer the way find_by.firstname("martin").and.lastname("fowler") puts field names into method calls and the data into parameters.

One of the problems with putting data into method names is that often, programming languages use a different encoding for program text than for their string data; many only allow ASCII, which then wouldn’t work for non-ASCII personal names. Similarly, the language grammar rules for method names may exclude valid personal names.

Above all, it’s important to remember that Dynamic Reception only pays its way when it allows you to build these structures in general, without any special case handling. This means that it’s only worthwhile when you can clearly translate from the dynamic methods to methods that are needed for other purposes. Conditions are good examples of this because they usually call attributes on domain model objects. A find_by_firstname_and_lastname method is effective because I have a Person class that has firstname and lastname attributes. If you need to write special methods to handle particular cases of Dynamic Reception, that usually means you shouldn’t be using Dynamic Reception.

Dynamic Reception comes with many problems and limitations. The biggest one, of course, is that you can’t do this at all with static languages. But even in a dynamic language, you need to be wary about using it. Once you override the handler for unknown method invocations, any mistake can lead you into a deep debugging trouble. Stack traces often become impenetrable.

There are also limitations on what you can express. You usually can’t put in something like find_by.age.greater_than.2 because most dynamic languages won’t allow “2” in a method name. You can dodge around that with something like find_by.age.greater_than.n2 but that obstructs much of the fluency that you’re doing this for.

Since I’m focusing on Boolean expressions here, I should also point out that this kind of method call composition is not a good way of composing complex Boolean conditions. The approach is fine for something simple like find_by.firstname("martin").and.lastname("fowler"), but once you get to statements like find_by.firstname.like("m*").and.age.greater_than(40).and.not.employer.like("thought*"), you’re running down a road that forces you to implement a kludgy parser in an environment not well-suited for it.

The fact that expressions using Dynamic Reception don’t work well for complex conditionals isn’t a reason to avoid them for simple cases. Active Record uses Dynamic Reception to provide dynamic finders for simple cases, but deliberately does not support more complex expressions, encouraging you to use a different mechanism instead. Some people don’t like that, preferring a single mechanism, but I think it’s good to realize that different solutions may work best at different complexities, so you should provide more than one.

41.3 Promotion Points Using Parsed Method Names (Ruby)

For this example, let’s consider a scheme that assigns points to travel itineraries. Our domain model is an itinerary that consists of items, where each item might be a flight, hotel stay, car hire, etc. We want to allow a flexible way for people to score frequent travel points, such as scoring 300 points for taking a flight out of Boston.

Using Dynamic Reception, I’ll show how to support the following cases. First is a simple case of one promotion rule.

@builder = PromotionBuilder.new
@builder.score(300).when_from("BOS")

In another case, we can have multiple promotion rules matching different kinds of elements. Here we score for flying out of a particular airport and staying an a particular hotel brand within the same itinerary:

@builder = PromotionBuilder.new
@builder.score(350).when_from("BOS")
@builder.score(100).when_brand("hyatt")

And finally, we have a compound flight rule, where we score for flying out of Boston on a particular airline (which may not be around any more when you read this).

@builder = PromotionBuilder.new
@builder.score(140).when_from_and_airline("BOS","NW")

41.3.1 Model

The model here has two parts: itineraries and promotions. The itinerary is just a collection of items, which could be anything. For this simple example, I just have flights and hotels.

Promotions are a set of rules, each rule having a score and a list of conditions.

The approach here scores an itinerary against a promotion. It does this by scoring each rule against the itinerary and summing up the results.

The rule scores an itinerary by checking if all its conditions match the itinerary; if so, it returns its score.

Each score line in the DSL is a separate rule. So,

@builder = PromotionBuilder.new
@builder.score(350).when_from("BOS")
@builder.score(100).when_brand("hyatt")

is one promotion with two rules. Either or both of them could match a given itinerary. In contrast,

@builder = PromotionBuilder.new
@builder.score(140).when_from_and_airline("BOS","NW")

is one rule with two conditions. Both conditions have to match to score the points.

To handle this, I have an equality condition object that I can set with appropriate names and values.

Using equality conditions in the method name like this is very limited. However, the underlying model allows me to have any kind of condition as long as it knows how to match an itinerary. Some of these conditions could be added through the DSL, others through other means, such as a closure.

This kind of flexibility can be quite important. It allows people to use the DSL to handle simple cases simply, but provides an alternative mechanism to handle more complicated cases.

41.3.2 Builder

The basic builder wraps a collection of promotion rules that it builds up, returning a new promotion object as needed.

The score method creates one of these rules, which it holds in a Context Variable. It also creates a particular builder for the condition.

The condition builder is the class that uses Dynamic Reception. In Ruby, you do Dynamic Reception by overriding method_missing.

The method_missing hook checks if the caller begins with when_; if not, it forwards to the superclass, which will throw an exception. Assuming we have the right kind of method, it pulls the attribute names out of the method call, checks that they match the arguments, and then creates the appropriate rules.

This approach is unsurprisingly similar to Active Record’s dynamic finders. If you’re curious about those, take a look at Jamis Buck’s description (http://weblog.jamisbuck.org/2006/12/1/under-the-hood-activerecord-base-find-part-3).

41.4 Promotion Points Using Chaining (Ruby)

Now I’ll take the same example and work it using chaining. I’ll use the same model and (mostly) the same example conditions. As the DSL is different, the conditions are formulated differently. Here’s the simple single selection of flights out of Boston:

@builder.score(300).when.from.equals.BOS

In this case, I’m passing all the arguments to the condition as methods rather than parameters (although I’m keeping the score as a parameter, just to be inconsistent). I’m also indicating the operator for the condition as a method.

Here’s the case with two separate scores:

@builder.score(350).when.from.equals.BOS
@builder.score(100).when.brand.equals.hyatt

Finally, here I have a compound condition:

@builder.score(170).when.from.equals.BOS.and.nights.at.least._3

The compound condition is more involved than the one I used in the previous example. For this one, I’m taking advantage of the ability to use other operators, as well as showing the kind of smudge you need to make in order to pass a numeric parameter as a method name.

41.4.1 Model

The model is almost identical to the one I used in the previous example. The only change is that I’ve added an extra condition.

41.4.2 Builder

The differences from the previous example lie in the builder. As before, I have a promotion builder object that holds a bunch of rules and produces a promotion when needed.

The score method adds a rule to the rules list.

The when method returns a more specific builder to capture the attribute name.

To build the condition, I create a little parse tree. Each condition in an expression has three parts: name, operator, and condition. So I make a builder for each of the parts, as well as a parent builder to tie the conditions together. As a result, when I create the name builder, I also create the condition builder parent to prepare the tree.

The attribute name builder will look for a suitable name for the attribute we are testing, since this name will vary depending on the model class’s attributes. I use Dynamic Reception.

This captures the name and returns the operator builder to capture the operator. The operator builder will only have a fixed set of operators to work off, so it doesn’t need to use Dynamic Reception.

The basic behavior of the operator builder is similar to the name builder: Capture the operator and return a new builder for the final part (the value). There are a couple of interesting points. Firstly, the content of this builder is the appropriate condition class from the model. Secondly, the at method just returns itself, as it’s pure syntactic sugar—only there to make the expression readable.

The final builder is the value builder which captures the value using Dynamic Reception.

If the value is a number, I need some jiggery-pokery, hence the use of a leading underscore “_3” to represent “3” in the DSL script. (In Ruby, "_3".to_i will parse the string to an integer, ignoring the underscore and returning 3.)

This method also ends this part of the expression, so it tells its parent to populate the model.

At this point, I’ve consumed the little parse tree and created the condition object in the model. If I have a compound condition, I repeat the process.

Making little parse trees like this isn’t a common way to do an internal DSL; it’s usually easier to just build the model up as you go. But with a conditional expression like this, it makes sense.

Overall, however, I’m not too keen on building up expressions using this approach. It seems to me that once you start parsing sequences of method calls like this, you might as well just switch to an external DSL where you get more flexibility. The desire to build up parse trees is a smell indicating that the internal DSL is doing too much work.

41.5 Removing Quoting in the Secret Panel Controller (JRuby)

In the introduction, I showed an example of how you could use Ruby as an internal DSL for the secret panel controller. The code looks like this:

In this example code, I don’t use any Dynamic Reception, relying on simple function calls. One of the disadvantages of this script is that there’s a lot of quoting, in particular every reference to an identifier needs Ruby’s symbol marker (the initial “:” in the names). Compared to an external DSL, this feels like noise. If I use Dynamic Reception, it’s possible to get rid of all of the symbol quoting and produce a script like this:

The starting point for implementing this is a state machine builder class. This class uses Object Scoping using instance_eval. The build occurs in two stages, first evaluating the script and then doing some postprocessing.

To evaluate the script, the builder has methods that correspond to the main clauses of the DSL script. I’m using the same Semantic Model that I used in the introduction; the JRuby builder populates Java objects.

The first clause to look at is the event declarations, which I make by calling the events method on the state machine builder, passing in a block that contains the individual event declarations.

The events method evaluates the block immediately in the context of a separate builder which uses Dynamic Reception to process every method call as an event declaration. With each event declaration, I create an event from the Semantic Model and put it into a Symbol Table.

I use the same basic technique for commands and reset events. By using a different builder, I can keep each one simple and clearly scope what each builder is recognizing.

The state declaration is more interesting. Again, I use a closure to capture the body of the declaration, but there are a couple of differences. The obvious one from the script is that I indicate the name with Dynamic Reception.

The second difference is in the implementation. Instead of evaluating the Nested Closure right away, I squirrel it away in a map. By deferring the evaluation till later, I can avoid worrying about the forward references between states. I can wait to deal with the state bodies until I’ve declared all the states and fully populated the Symbol Table with them.

The last point is that I treat the first-named state as the start state by using an additional variable which I populate only if it’s still nil—which means that the first state will be in there.

Populating this data finishes evaluating the script; now, the second stage is the postprocessing.

The first step of postprocessing is to evaluate the bodies of the state declarations, again by creating a specific builder and instance_evaling the block with it.

The body can contain two kinds of statements: declaring actions and declaring transitions. The action case is handled by a specific method.

The actions creates another builder which absorbs all method calls as command names. This allows you to specify multiple actions in a single line with chaining.

While actions use a special method, analogous to a keyword in an external DSL, transitions use Dynamic Reception.

Here I use an unknown method to start a specific builder to capture the target state with a further use of Dynamic Reception. I also allow to as syntactic sugar.

By doing all of this, I can get rid of all the “:” on symbols. The question, of course, is whether it’s worth the trouble. To my eye, I like the way the event and command list turn out, but I’m not so keen on the states. I could, of course, use a hybrid approach with Dynamic Reception for the things I like and symbol references where Dynamic Reception isn’t helping. A mixture of techniques is often the best bet.

Getting rid of the symbol “:” is nice, but I still have the quotes around command and event codes. I could use a similar technique to deal with those as well.