Chapter 18. Convention Over Configuration
As the final pattern discussed in this book, we will look at Convention Over Configuration, a pattern that did not originate in Design Patterns but comes to us straight from the Rails framework. Convention Over Configuration is arguably one of the keys to the success of Rails. It is a bit different from the other patterns that we have examined in this book in that it is bigger and more ambitious. While the other patterns mostly dealt on the smaller scale of pulling together a number of related classes, Convention Over Configuration is concerned with pulling together whole applications and application frameworks. How do you structure an application or a framework so that it is extensible, so that other engineers can easily add bits to it as the program evolves over time? As we construct ever more ambitious systems, the problem of making them configurable and extensible looms ever larger.
The reaction of the software world to the problem of extensibility reminds me of a quandary I faced when I was in elementary school. You see, I could never decide when to do my homework. Some days I would run home from school, throw open those books, and just get it over with. There is nothing like the feeling of just being done with it. But there is also nothing like coming home, chucking the books on the dining room table, jumping on a bike, and heading out for driveways afar. Of course, those stinking books would still be there—and eventually I would have to return, sweaty and tired, and do the homework anyway. Ultimately, I reached a compromise with myself: Do the hated English and boring social studies right after school, and leave the easy math until later in the evening.
Software engineering has gone through an analogous process when it comes to making our systems extensible. Many of us grew up professionally with applications that took pride in their own limitations. These programs supported exactly one protocol, or required that the database schema look just so, or imposed some inflexible interface on hapless users.
The reaction to the heartache produced by this rigidity was to make software systems extensible via configuration. If we pushed the important decisions into a configuration file, we could drive our system to the Utopian ideal of "Hey, just configure me to do what you want." Sadly, we kept right on driving—through downtown Utopia, past the Utopian suburbs, and out into the far side into a new wasteland. We now have configuration-dependent code. We are afflicted with frameworks that are afraid to make the slightest configuration commitment and applications that live in fear that any assumption will inevitably need to be overturned in a hastily arranged patch release.
Java servlets provide a good example of too much configuration. Servlets are a key component of virtually all Java Web applications. A servlet is an elegant little Java class that knows how to handle HTTP requests coming into one or more URLs. But writing a Java class that extends javax.servlet.HttpServlet is not enough—not nearly enough. No, you also need to configure the thing, via the web.xml configuration file. In its most basic form, a web.xml lets you associate a servlet class with an arbitrary name and then associate that arbitrary name with one or more URLs.
And yet, in real life, we rarely need all this flexibility. Mostly—not always, but mostly—we tend to use the class name as the arbitrary name. Mostly—not always, but mostly—we associate the arbitrary name with a URL that bears a startling resemblance to the arbitrary name and, by extension, to the class name. We do so because any name will suffice: best use one that will remind us of the class. And we usually do not care about the exact URL that is fronting our servlet: best use one that reminds us of the class behind the servlet. Programmers (or the good ones, anyway) value simplicity, and the simple thing to do here is to rigorously cancel out all of that flexibility. If you have no use for it, flexibility becomes a danger. All those names and associations become just another way to screw up.
The main motive behind the Convention Over Configuration pattern is to lighten that configuration burden. We want to preserve the essential extensibility of our applications and frameworks, yet get rid of the extraneous configuration. In this chapter we will begin by looking at the principles behind the Convention Over Configuration pattern. Then we will build a hypothetical messaging system showing how you can construct software that, like my final homework strategy, stands in the happy middle ground.
A Good User Interface—for Developers
The problem of writing software that is both flexible and easy to use is a familiar one. Well, it’s familiar if you build graphical user interfaces (GUIs) for a living. The folks who build GUIs have evolved a number of design guidelines for creating easy-to-use interfaces:
- Attempt to anticipate the user’s needs. A good interface tries to make very common tasks nearly effortless—a good interface does the most common case by default. Uncommon or more advanced tasks should still be doable with a bit more effort.
- Don’t make the user repeat himself or herself. Who among us has not been tempted to put a foot through the CRT when that application asks, for the third time, "Are you sure you want to do this?"
- Provide a starter template. Providing your user with a template to build on is another one of those good GUI ideas that we can port to our systems. Don’t make your user start with a blank sheet—if he is creating a résumé, give him a résumé template to get him started.
The Convention Over Configuration pattern focuses on applying these same principles to the design of applications and framework APIs. Why save all of the good techniques for the end user? The engineers who are trying to configure your application or program to your API are users, too, and they could use a hand as well. Why not provide a good interface for all your users?
Anticipate Needs
No matter whether we are talking about the GUI for an e-mail client or the design of an API, there is one thing that sets a good user interface apart from a bad one: If the user wants to do it (whatever it is) a lot, it is the default. Conversely, if the user does not want to do it all that often, it can be a bit harder. This is the reason why moving to the next message in my in-box takes only a press of the down arrow key, whereas doing the configuration for a new e-mail server requires navigating through several menus.
All too often we build APIs that assume every action is exactly as likely as every other action. You can see this assumption in Java servlet configuration—it doesn’t matter if I am creating a very commonplace servlet that answers to exactly one URL or a more complex, multipurpose servlet that is hooked up to a number of URLs. It doesn’t matter because the more common task takes just about the same amount of work as the less common task. A more considerate interface would make the more common case easy, while requiring somewhat more work for the less common case.
Let Them Say It Once
Another way to drive your technical users crazy is by forcing them to repeat themselves. We know this when it comes to the traditional user interfaces (the kind with menus and icons), but for some reason we tend not to apply the same logic to APIs. How can we avoid making our technical users repeat themselves? We can give them a way to tell us what they want and not ask again. Engineers naturally tend to adopt conventions as a natural part of the way they work. They tend to name files and classes a certain way, to group source files that do similar things in the same directory, and to christen methods with names that follow regular patterns.
The Convention Over Configuration pattern suggests that you define a convention that a sensible engineer might use anyway—put all of the adapters in this directory or name all of the authorization methods this way—and then run with it. Designing a good convention, like designing any good user interface, involves putting yourself in your user’s shoes. Try to deduce how your users will behave, what they would call something, and where they would naturally put things; then build your convention around those assumptions. Once you have that convention in hand, get as much mileage out of it as you possibly can—by naming his or her class or putting it in a given directory, the engineer is telling you something. Listen and don’t make the engineer tell you again.
Provide a Template
Another thing that you can do is to give your user a kick start by supplying him or her with a model, a template, or an example to follow. Modern word processors no longer expect you to start with a completely blank sheet of electronic paper. Instead, when you create a new document, the program wants to know if it is a résumé, a letter, or a presidential speech. If your document is any one of those or another hundred document types, a good word processor will start you off with the right margins and paragraph styles.
You can do the same for the people who are trying to extend your system: You can give them samples, templates, and working examples to help them get off the ground. If a picture is worth a thousand words, then one or two good examples have got to be worth at least twenty pages of documentation.
A Message Gateway
To see how we can apply these lofty ideals to real code, let’s imagine that we have been asked to build a message gateway. Our code will have the job of receiving messages and then sending them on to their final destinations. Messages look like this:
require 'uri'
class Message
attr_accessor :from, :to, :body
def initialize(from, to, body)
@from = from
@to = URI.parse(to)
@body = body
end
end
The from field is a simple string containing something like 'russ.olsen', indicating who is sending the message. The to field is a URI telling us where we should send the message. The body field is a string holding the actual contents of the message. The Message class uses the URI class, which is a standard part of your Ruby installation, to turn the to string into a useful URI object. Initially, the to URIs will come to the gateway in three flavors. You will need to send the message out as an e-mail:
smtp://[email protected]
Or via an HTTP Post request:
http://russolsen.com/some/place
Or to a file:
file:///home/messages/message84.txt
A key requirement of our message gateway is that it should be easy to add new protocols. For example, if we need to send messages via FTP, it should be very easy to extend the gateway to handle the new destinations.
Looking through your favorite book on design patterns,[1] you realize that what you need to handle these different message destinations is an adapter. More precisely, you need three adapters, one for each protocol. In this case, the adapter interface is very simple, consisting of just a single send_message(message) method. Here is the adapter that handles forwarding the message as e-mail:
require 'net/smtp'
class SmtpAdapter
MailServerHost = 'localhost'
MailServerPort = 25
def send_message(message)
from_address = message.from.user + '@' + message.from.host
to_address = message.to.user + '@' + message.to.host
email_text = "From: #{from_address}
"
email_text += "To: #{to_address}
"
email_text += "Subject: Forwarded message
"
email_text += "
"
email_text += message.body
Net::SMTP.start(MailServerHost, MailServerPort) do |smtp|
smtp.send_message(email_text, from_address, to_address)
end
end
end
Here is the adapter that uses HTTP to send the message on its way:
require 'net/http'
class HttpAdapter
def send_message(message)
Net::HTTP.start(message.to.host, message.to.port) do |http|
http.post(message.to.path, message.body)
end
end
end
Finally, here is the adapter that "sends" the message by copying it to a file:
class FileAdapter
def send_message(message)
#
# Get the path from the URL
# and remove the leading '/'
#
to_path = message.to.path
to_path.slice!(0)
File.open(to_path, 'w') do |f|
f.write(message.body)
end
end
end
Picking an Adapter
The next problem that you have is matching up a message with the proper adapter class that will send that message on its way. One solution is to hard-code the adapter selection logic:
def adapter_for(message)
protocol = message.to.scheme
return FileAdapter.new if protocol == 'file'
return HttpAdapter.new if protocol == 'http'
return SmtpAdapter.new if protocol == 'smtp'
nil
end
This hard-coding solution has a problem, however: Anyone adding a new delivery protocol—and therefore a new adapter—would have to dive into the adapter_for method to add another adapter. Making someone change existing code does not seem to fall within the bounds of "easily extensible." Perhaps we can do better. Maybe we should have a configuration file that maps protocols to adapter names, something like this:
smtp: SmtpAdapter
file: FileAdapter
http: HttpAdapter
We could also make this solution work, but with the configuration file we have just traded one form of hard-coding for another. Either way, the person who is adding a new adapter not only needs to write the adapter class, but must also do something else to get the system to recognize the new adapter.
This brings us to the punch line: Why not make writing the adapter class all that is required? If we ask the adapter writer to adhere to the following, very sensible convention, we can reduce the job of adding a new adapter to simply writing the adapter class. Here is the magic convention:
Name your adapter class <protocol>Adapter.
Following this convention, a new adapter to send files via FTP would be called FtpAdapter. If all of the adapters follow this convention, then the system can pick the adapter class based on its name:[2]
def adapter_for(message)
protocol = message.to.scheme.downcase
adapter_name = "#{protocol.capitalize}Adapter"
adapter_class = self.class.const_get(adapter_name)
adapter_class.new
end
The adapter_for method pulls the destination protocol off of the message and, using a bit of string legerdemain, transforms a name like 'http' into 'HttpAdapter'. From there it is a matter of a call to const_get to get the class of the same name. With this approach, we have completely lost any hint of a configuration file—to add a new adapter, you simply add the adapter class.
Loading the Classes
Well, almost. We still have to deal with the fact that we need to load the adapter classes into the Ruby interpreter. In terms of code, we need to require in the files that contain the adapter classes:
require 'file_adapter'
require 'http_adapter'
require 'smtp_adapter'
Now we could simply put all of the adapter require statements in a file and tell the adapter writer to be sure to add his or her adapter to the list. But once again, we are asking the adapter writer to repeat himself or herself; we are asking the adapter writer to tell us twice that an adapter exists—once by writing and properly naming the adapter class and again by adding it to the list of include statements. Besides, we can do better.
Let’s start doing better by focusing on directory structures. While we tend to ignore the files and directories that play host to our software, we can actually get a lot of mileage out of conventions based on where various files live. Imagine that we set up a directory structure for our gateway system along the lines of that shown in Figure 18-1.
Figure 18-1. Directory structure for the message gateway system
This directory structure is not particularly original—it is a very common layout for Ruby projects.[3] It is particularly germane to the question of finding adapters because we can use a standard directory structure to solve the problem of loading adapters:
def load_adapters
lib_dir = File.dirname(__FILE__)
full_pattern = File.join(lib_dir, 'adapter', '*.rb')
Dir.glob(full_pattern).each {|file| require file }
end
The load_adapters method computes the path of the adapter directory by starting with the __FILE__ constant, which the Ruby interpreter always sets to the path to the current source file. A little manipulation using various methods from the File class allows us to come up with a filename pattern for all of our adapters—something like "adapter/*.rb" . The method then uses this pattern to find and require in all of the adapter classes. This scheme works because require is just another method call in Ruby; we can call it from code whenever we need to pull a source file into the Ruby interpreter. We do need to beef up our adapter convention a bit, though:
Name your adapter class <protocol>Adapter and put it in the adapter directory.
Remarkably, this is pretty much all we need to get a very basic message gateway working. Here is the MessageGateway class in full:
class MessageGateway
def initialize
load_adapters
end
def process_message(message)
adapter = adapter_for(message)
adapter.send_message(message)
end
def adapter_for(message)
protocol = message.to.scheme
adapter_class = protocol.capitalize + 'Adapter'
adapter_class = self.class.const_get(adapter_class)
adapter_class.new
end
def load_adapters
lib_dir = File.dirname(__FILE__)
full_pattern = File.join(lib_dir, 'adapter', '*.rb')
Dir.glob(full_pattern).each {|file| require file }
end
end
Call the process_message method and pass in a Message object, and in no time your missive will be winging its merry way to its destination.
There are two things to note about our adapter convention. First, the convention focuses squarely on making it easy to add adapters. We did not try to make the whole message gateway easily extensible in every dimension; we just tried to make it easy to add new adapters. Why? Because we anticipate that adding new adapters is what our users (i.e., the future adapter-writing engineers) will need to do. By anticipating that need, we can make life easier for the adapter writer.
Second, while our convention does impose some constraints on the adapter writer—he or she needs to name the adapter just so and put it into the correct directory—these constraints are really quite innocuous. They are, in fact, exactly the kind of thing that a careful engineer would do anyway.
Adding Some Security
Now that we have the basic message gateway working, we need to think about adding some security. Specifically, we want to apply a separate policy to control which users are allowed to send messages to a given host. In addition to the general policy, we need to deal with a variety of special users, who are exceptions to the general policy for a given host.
We can start by deciding to have one authorization class per destination host; this constraint seems acceptable as long as we are dealing with a limited number of hosts. The directory and class name-based convention worked so well for the adapters that we will adopt a similar one for authorization classes:
Name your authorization class <destination_host>Authorizer and put it in the auth directory.
Figure 18-2 shows our updated directory structure.
Figure 18-2. Gateway directory structure extended for authorization
Naming the authorization classes after hosts does bring up a problem, however: Host names generally are not valid Ruby class names. What we need is a little string transformation magic. Let’s translate a host name like russolsen.com to the authorization class RussolsenDotComAuthorizer:[4]
def camel_case(string)
tokens = string.split('.')
tokens.map! {|t| t.capitalize}
tokens.join('Dot')
end
def authorizer_for(message)
to_host = message.to.host || 'default'
authorizer_class = camel_case(to_host) + "Authorizer"
authorizer_class = self.class.const_get(authorizer_class)
authorizer_class.new
end
But what should the interface for the authorization classes look like? Recall that for any given host, there should be a single set of rules that applies to almost all users. I say "almost" because there may be an exceptional user or two who obey their own rules. For example, we might imagine that anyone is allowed to send short messages to russolsen.com, but only 'russ.olsen' himself is allowed to send really long messages.
One convention we might adopt is that if an authorization class has a method called <user name>_authorized?, then we will use that method to authorize the message. Of course, we will have to suitably transform the user name to fit the rules of a method name. If there is no such method, we will fall back to using a generic authorized? method. A typical authorizer class might look like this:
class RussolsenDotComAuthorizer
def russ_dot_olsen_authorized?(message)
true
end
def authorized?(message)
message.body.size < 2048
end
end
The code to implement this policy convention is very straightforward. First, we get an instance of the authorizer for the message that we are processing. Next, we work out the name of the special policy method for the user who sent the message. Finally, we check whether our authorizer object answers to that method. If it does, then that is the method we use. If not, then we use the standard authorized? method.
def worm_case(string)
tokens = string.split('.')
tokens.map! {|t| t.downcase}
tokens.join('_dot_')
end
def authorized?(message)
authorizer = authorizer_for(message)
user_method = worm_case(message.from) + '_authorized?'
if authorizer.respond_to?(user_method)
return authorizer.send(user_method, message)
end
authorizer.authorized?(message)
end
Here is our final authorization convention:
Name your authorization class <destination_host>Authorizer and put it in the auth directory. Implement the general policy for the host in the authorize method. If you have a special policy for a given user, implement that policy in a method called <user>_authorized?.
Getting the User Started
There is one other way that we can help the engineer who needs to extend the gateway—we can help him or her get started. Earlier in this chapter, we noted that one of the principles of good interface design is to provide the user with templates and samples to help get started. At one end of the spectrum, we might provide a few examples showing how to create a protocol adapter or an authorizer. At the other end of the spectrum, we could supply a utility to generate the outline or scaffold of a class.
For example, we might supply the prospective adapter writer with the following Ruby script, which will generate the bare bones of an adapter class:
protocol_name = ARGV[0]
class_name = protocol_name.capitalize + 'Adapter'
file_name = File.join('adapter', protocol_name + '.rb')
scaffolding = %Q{
class #{class_name}
def send_message(message)
# Code to send the message
end
end
}
File.open(file_name, 'w') do |f|
f.write(scaffolding)
end
If we put this code in a file called adapter_scaffold.rb, then we can use it to generate a starter adapter for FTP by running
ruby adapter_scaffold.rb ftp
We end up with a class called FtpAdapter in a file called ftp.rb in the adapter directory.
It is easy to discount the value of this scaffold-generating script. Nevertheless, these kinds of utilities are invaluable to the new user who is overloaded with information about an unfamiliar application or environment, and is struggling just to get started.
Taking Stock of the Message Gateway
We could continue to extend our message gateway, perhaps by adding a transformation step to reformat the message or a flexible auditing facility to log some but not all of the messages. But let’s stop here and consider what we have accomplished. We have built a message gateway that is extensible in two different dimensions: You can support new protocols and you can add new authorization policies, complete with exceptions for individual users. We accomplished this with absolutely no configuration files. Instead, the would-be system extender simply needs to write the correct class and drop it into proper directory.
An interesting and unexpected side effect of using conventions is that this approach actually simplifies the main gateway code itself. If we had used configuration files, we would have had to locate and read in the files, perhaps check for errors, and only then begin to set up our adapters and authorization classes. Instead, we simply got on with the task of finding our adapters and authorizers.
Using and Abusing the Convention Over Configuration Pattern
One danger in building convention-based systems is that your convention might be incomplete, thereby limiting the range of things that your system can do. Our message gateway, for example, does not really do a thorough job of transforming host names into Ruby class names. The code in this chapter will work fine with a simple host name like russolsen.com, transforming it into RussOlsenDotCom. But feed our current system something like icl-gis.com and it will go looking for the very illegal Icl-gisDotComAuthorizer class. You can usually solve this kind of problem neatly by allowing classes to override the conventions when necessary. In our example, we could allow each authorization class to potentially override the default host name-mapping behavior and specify the hosts to which it applies.
Another potential source of trouble is the possibility that a system that uses a lot of conventions may seem like it is operating by magic to the new user. Configuration files may be a pain in the neck to write and maintain, but they do provide a sort of road map—perhaps a very complicated and hard-to-interpret road map, but a map nevertheless—to the inner workings of the system. A well-done convention-based system, by contrast, needs to supply its operational road map in the form of (gasp!) documentation.
Also keep in mind that as the convention magic becomes deeper and more complex, you will need ever more thorough unit tests to ensure that your conventions behave, well, conventionally. There are few things more confusing to users than a system driven by an inconsistent or just plain broken set of conventions.
Convention Over Configuration in the Wild
Rails is still the best example of a system sewn together by conventions. Certainly, our message gateway example lifts many of the Rails convention ideas whole cloth. Indeed, you can trace much of the elegance of Rails to its consistent use of conventions:
- If your Rails application is deployed on http://russolsen.com, then a request on http://russolsen.com/employee/delete/1234 is handled by default by a call to the delete method on the EmployeeController class. The 1234 is passed in to the method as a parameter.
- The results of that controller call are, by default, handled by the view defined in the views/employee/delete.rhtml file.
- Rails applications typically use ActiveRecord to talk to the database. By default, a table called proposals (note the plural) will be handled by a class called Proposal (singular) that lives in a file called proposal.rb (note the lowercase) that lives in the models directory. A field called comment in the proposals table shows up unassisted as a field called comment in the Proposal object.
- Rails comes complete with a whole set of scaffold-generating utilities that help the user create a starter model, views, and controllers.
A typical Rails application is literally sewn together by conventions of one sort or another.
But Rails is not the only example of the wise application of conventions in the Ruby world. RubyGems is the standard software packaging utility used by Ruby applications. It is relatively easy to use, especially if you follow its directory layout conventions—as we did with the message gateway.
Wrapping Up
In this chapter, we looked at the Convention Over Configuration pattern. Convention Over Configuration says that you can sometimes build a friendlier system by binding your code together using conventions based on class names, method names, filenames, and a standard directory layout. By doing so, you can make your programs easily extensible; you can extend your system by simply adding in a properly named file or class or method.
The Convention Over Configuration pattern takes advantage of the same dynamic and flexible Ruby features that make the other two Ruby-specific patterns that we examined in this book possible. Like the Domain-Specific Language pattern, Convention Over Configuration relies heavily on runtime evaluation of code. Like the Meta-programming pattern, it requires a fairly high level of program introspection to function. But these three patterns all share something else: an approach to solving programming problems. Their common message is that you should not just take your language as you find it, but rather mold it into something closer to the tool that you need to solve the problem at hand.