Intent

To create a miniature programming language tailored to solve a specific problem

Overview

Domain-Specific Language is a very common pattern that has two broad classes: external DSL and internal DSL.

An external DSL is a full-blown programming language with its own syntax and compiler. It’s not intended for general use; rather, it solves some targeted problems. For instance, SQL is an instance of Domain-Specific Language targeted at data manipulation. ANTLR is another, targeted at creating parsers.

On the other hand, we’ve got internal DSLs, also known as embedded languages. These instances of the pattern piggyback on some general-purpose language and live within the constraints of the host language’s syntax.

In both cases, the intent is the same. We’re trying to create a language that lets us express solutions to problems in a way that is closer to the domain at hand. This results in less code and clearer solutions than those created in a general-purpose language. It also often allows people who aren’t software developers to solve some domain problems.

In this section, we’ll look at building internal DSLs in Scala and Clojure. The techniques we’ll use to build a DSL are very different in these two languages, but the intent remains the same.

In Scala

The current crop of Scala DSLs rely on Scala’s flexible syntax and several other Scala tricks. The Scala DSL we examine here will take advantage of several of Scala’s advanced abilities.

First off, we’ll see Scala’s ability to use methods in the postfix and infix positions. This lets us define methods that act as operators.

Second, we’ll use Scala’s implicit conversions, introduced in Pattern 10, ​Replacing Visitor​. These appear to let us add new behavior to existing types.

Finally, we’ll use a Scala companion object as a factory for the class it’s paired up with.

In Clojure

Internal DSLs are an old Lisp technique that Clojure carries on. In Clojure and other Lisps, the line between Domain-Specific Language and frameworks or APIs is very blurry.

Good Clojure code is often structured as layers of DSLs, one on top of the other, each of which is good at solving a problem on a particular layer of the system.

For example, one possible layered system for building web applications in Clojure starts with a library called Ring. This provides an abstraction over HTTP, turning HTTP requests into Clojure maps. On top of that, we can use a DSL named Compojure to route HTTP requests to handler functions. Finally, we can use a DSL named Enlive to create templates for our pages.

Clojure’s DSLs are generally built around a core set of higher-order functions, with macros providing syntactic sugar on top. This is the approach we’ll use for the Clojure DSL we examine here.

Code Sample: DSL for a Shell

I sometimes find myself cutting and pasting from a shell into a REPL when programming in Scala and Clojure. Let’s take a look at a simple DSL to make this more natural by letting us run shell commands directly in a REPL.

In addition to running commands, we’ll want to capture their exit status, standard output, and standard error. Finally, we’ll want to pipe commands together, just as we can in a normal shell.

In Scala

The end goal of this example is to be able to run shell commands in a natural way inside of a Scala REPL. For individual commands, we’d like to be able to run them like this:

And we’d like to run pipes of commands like so:

Let’s take our first step on our shell DSL journey by examining what we want a command to return. We need to be able to inspect a shell command’s status code and both its standard output and error streams. In the following code, we packaged those pieces of information together into a case class named CommandResult:

Now let’s see how to actually run a command. We can dip into Java’s ProcessBuilder class for this.

The ProcessBuilder class constructor takes a variable number of string arguments, representing the command to run and its arguments. In the following REPL snippet, we create a ProcessBuilder that will allow us to run ls -la :

To run the process, we call start on the ProcessBuilder we just created. This returns a Process object that gives us a handle on the running process:

The Process object gives us access to all the information we need, but output from standard out and standard error are inside of InputStream objects rather than inside strings. We can use the fromInputStream on Scala’s Source object to pick them out, as we demonstrate in the following code:

Notice how the method that gets us the output from standard out is somewhat confusingly called getInputStream()? That’s not a typo; the method name seems to refer to the fact that standard out is being written into a Java InputStream that the calling code can consume.

Now we can put our Command class together. The Command takes a list of strings representing the command and its arguments and uses it to construct a ProcessBuilder. It then runs the process, waits for it to complete, and picks out the completed process’s output streams and status code. The following code implements Command:

To make Command classes a bit easier to construct, we add a factory method that takes a string and splits it into Command’s companion object:

As the following REPL session demonstrates, this gets us a bit closer to our desired syntax for running a single command:

To get the rest of the way there, we’ll use the implicit conversions we introduced in Pattern 10, ​Replacing Visitor​. We’ll create a conversion that turns a String into a CommandString with a run method. A CommandString turns the String it’s converting into a Command that its run method calls. It’s implemented in the following code:

Now we’ve got our desired syntax for running single commands, as we demonstrate with the following REPL output:

Let’s extend our DSL to include pipes. The approach we’ll take is to collect our piped command strings into a vector and run them once we’ve constructed the full chain of pipes.

Let’s start off by examining the extensions we need to make to CommandString. Remember, we’d like to be able to run a pipe of commands like so: "ls -la" pipe "grep build" run. This means we need to add a pipe method, which takes a single string argument, to our CommandString implicit conversion. When it’s called, it’ll take the string it’s converted to a CommandString and the argument it was passed, and it’ll stuff them both into a Vector. The code for our expanded CommandString follows:

Now our conversion will convert "ls -la" pipe "grep build" into a vector with both shell commands in it.

The next step is to add another implicit conversion that converts a Vector[String] into a CommandVector, much as we’ve already done for individual strings. The CommandVector class had a run and a pipe method.

The pipe method adds a new command to the Vector of commands and returns it, and the run method knows how to go through the commands and run them, piping the output from one to the next. The code for CommandVector and a new factory method on the Command companion object used by CommandVector follows:

Now we’ve got our full DSL, pipes and all! In the following REPL session, we use it to run some piped commands:

A couple of notes on this DSL. First, it takes advantage of Scala’s ability to use methods as postfix operators. This is easy to misuse, so Scala 2.10 generates a warning when you do so, and it will be disabled by default in a future version of Scala. To use postfix operators without the warning, you can import scala.language.postfixOps into the file that needs them.

Second is a simple DSL, suitable for basic use at a Scala REPL. Scala has a much more complete version of a similar DSL already built into the scala.sys.process package.

In Clojure

In Clojure, our DSL will consist of a command function that creates a function that executes a shell command. Then we’ll create a pipe function that allows us to pipe several commands together using function composition. Finally, we’ll create two macros, def-command and def-pipe, to make it easy to name pipes and commands.

Before we jump into the main DSL code, let’s take a look at how we’ll interact with the shell. We’ll use a library built into Clojure in the clojure.java.shell namespace, which provides a thin wrapper around Java’s Runtime.exec().

In the following REPL session, we use the sh function in clojure.java.shell to execute the ls command. As we can see, the output of the function is a map consisting of the status code for the process and whatever the process wrote to its standard out and standard error streams as a string:

This isn’t very easy to read, so let’s create a function that’ll print it in a way that’s easier to read before returning the output map. The code to do so follows:

We can now use sh to run ls -a and get readable output:

Let’s move on to the first piece of our DSL, command function. This function takes the command we want to execute as a string, splits it on whitespace to get a sequence of command parts, and then uses apply to apply the sh function to the sequence.

Finally, it runs the returned output through our print-output function, wraps everything up in a higher-order function, and returns it. The code for command follows:

Now if we run a function returned by command, it’ll run the shell command it encapsulates:

If we want to name the command, we can do so using def:

Now that we can run an individual command, let’s take a look at what it’ll take to pipe them together. A pipe in a Unix shell pipes the output from one command to the input of another. Since the output of a command here is captured in a string, all we need is a way to use that string as input to another command.

The sh function allows us to do so with the :in option:

Let’s modify our command function to take the output map from another command and use its standard output string as input. To do so, we’ll add a second arity to command that expects to be passed an output map.

The command function destructures the map to pluck out its output and passes it into sh as input. The code for our new command follows:

Now we can define another command, like the following one that greps for the word README:

Then we can pass the output of our ls command into it, and the ls output will be piped into grep. Each command will print its output to standard out, as the following REPL session shows:

With our modified command function, we can create a pipe of commands by composing together several commands with comp. If we want to write the commands in the same order as we would in a shell, we just need to reverse the sequence of commands before we compose them, as we do in the following pipe implementation:

Now we can create a pipe of commands, as we do in the following REPL session:

This has the same effect as running the ls command and passing its output into grep-readme:

Now that we can define commands and pipes, let’s use macros to add some syntactic sugar to make things easier. For an introduction to Clojure’s macros, see ​Clojure Macros​. First we’ll create a def-command macro. This macro takes a name and a command string and defines a function that executes the command string. The code for def-command follows:

Now we can define a command and name it with a single macro invocation, as we do in the following REPL output:

Now let’s do the same for our piped commands as we did for single commands with the def-pipe macro. This macro takes a command name and a variable number of command strings, turns each command string into a command, and finally creates a pipe with the given name. Here’s the code for def-pipe:

Now we can create a pipe in one shot, as we do below:

That wraps up our look at Clojure’s DSLs!

Discussion

Currently, Scala and Clojure take a very different approach to Domain-Specific Language. Scala uses a flexible syntax and a variety of tricks. Clojure uses higher-order functions and macros.

Clojure’s approach is more general. In fact, most of the Clojure language itself is written as a set of Clojure functions and macros! Advanced Scala DSL writers may bang up against the limitations of Scala’s current approach.

For this reason, macros are being added to Scala. However, as noted in the ​Discussion​, they’re much harder to implement and use without the simple syntax and homoiconicity available in Clojure and other languages in the Lisp family.

Related Patterns

Pattern 20, ​Customized Control Flow​

For Further Reading

DSLs in Action [Gho10]