Chapter 9. DSL design: looking forward

This chapter covers

• Overview of our journey together
• Expanding support for DSL development
• Increasing tool support for writing DSLs
• The continuing evolution of DSLs

Congratulations! You’ve reached the last chapter of the book. We’ve covered a lot of ground as I’ve told you about the paradigms of DSL-based development. We’ve discussed all the aspects of DSL design using quite a few languages, mostly on the JVM. I carefully chose a good mix of statically and dynamically typed languages, covering both OO and functional paradigms. In this chapter, we’ll look at trends in DSL development that are becoming more popular and mainstream. As a practitioner, you need to be aware of these developments; some might eventually evolve and mature into useful implementation techniques.

We’re going to discuss some of the areas that DSL designers are focusing on that enhance DSL-based development. Figure 9.1 shows a roadmap of the features that you’ll learn about on the last leg of our journey together.

Figure 9.1. Our roadmap through the chapter

We’re going to start in section 9.1 with language expressivity, where the horizon seems to be expanding by the day. Groovy, Ruby, Scala, and Clojure are way more expressive than Java and have been continuously evolving, while still keeping a human interface in mind. Of these languages, the dynamic ones like Groovy, Ruby, and Clojure already use the power of metaprogramming, which is even being added in a couple of statically typed languages. Even if you don’t use metaprogramming now, as an expert in DSL development, you need to keep yourself up to date about how these features are shaping today’s languages to create a better environment for DSL construction.

Another addition to the DSL implementation technique is the use of parser combinators. More languages that have functional programming capabilities will gradually offer libraries that implement parser combinators as part of their standard distribution. We’re going to discuss this in section 9.1.

Next we’ll step into something that has the potential to be a new norm in DSL development: the use of DSL workbenches. Section 9.3 discusses additional tool support in modern IDEs. I conclude the chapter with a section about DSL evolution where I’ll discuss how to grow a DSL in a disciplined way, keeping an eye on the language’s backward compatibility.

If in the earlier chapters you learned about the present of DSL design, in this chapter you’ll get an idea of what to look for in the future of DSLs. You need to be prepared for tomorrow, because today is already here.

9.1. Growing language support for DSL design

DSLs are all about being expressive with respect to the domain that you model. When you’re modeling an accounting system, you want your API set to speak in terms of debits, credits, books, ledgers, and journals. But these are only the nouns of the model, the concrete artifacts that form some of the core concepts in the problem domain. You also need to convey the verbs with the same level of expressiveness that you speak in the problem space. Remember our coffee shop example in chapter 1 that started us off? The barista could serve precisely what you ordered because you spoke in the language she understands. The whole expression needs to have a synergy with the problem domain that’s being modeled. Consider the following snippet of Scala code that we discussed in chapter 3:

withAccount(trade) {
  account => {
    settle(
      trade using
        account.getClient
               .getStandingRules
               .filter(_.account == account)
               .first)
    andThen journalize
  }
}

This snippet is a DSL from the domain of securities trading system. Note how the nouns and verbs of the domain are expressively modeled through the DSL syntax. Scala’s support for higher-order functions lets you treat your domain behaviors (the verbs) as uniformly as the domain objects (the nouns). This uniform treatment makes the language expressive.

You might be wondering why I’m being so emphatic about something in the last chapter of the book when the same subject has been the underlying theme throughout our discussion. I feel the need to reemphasize the fact that for a sufficiently powerful language, the expressiveness is limited only by the creativity of its users. Powerful idioms like metaprogramming, functional control structures, and a flexible enough type system let a programmer express the problem domain in a DSL that speaks the same language as the domain itself. In this section, we’ll discuss how some of today’s languages are extending the frontiers of expressivity to position themselves as potent forces for DSL development.

9.1.1. Striving to be expressive

With newer languages coming up pretty quickly, we’ll be seeing more and more support that makes for expressive syntax design for your DSL. In the earlier chapters, we discussed all the power that Ruby, Groovy, Scala, and Clojure offer in this respect. In this section, I’ll give you a brief overview of the capabilities that some of the other languages offer. I won’t discuss them in detail; the main idea is to show you that now there are more languages that strive to be expressive to the human interface. Look at figure 9.2, which shows the progression of how some mainstream languages have evolved into more expressive ones in the course of time.

Figure 9.2. Evolution of expressiveness in programming languages

Expressive programming languages help close the semantic gap between the problem domain and the solution domain. In OO languages that don’t support higher-order functions, you need to shoehorn objects as functors to model domain actions. Obviously this indirection manifests itself as accidental complexity (see appendix A) in the resulting DSL that you design. With support of first-class functions, your DSL becomes much cleaner and more acceptable to your users.

Now we’re going to look at how DSL development practices have evolved over the years with increasing expressiveness of programming languages. Remember that we used to write domain rules in the days of C as well, only at a much lower level of abstraction than what we do today. Figure 9.3 shows this progression in DSL development.

Figure 9.3. Evolution of the features in programming languages we use to develop DSLs

Many of these features are mature today, but others are still evolving and being adopted by more and more languages. I’m going to describe three of the important features that have been gaining more ground in the ecosystem of DSL-based development. I’ll start with a technique that’s already become popular in the dynamic languages. Given its potential use in DSL design, metaprogramming is being introduced in more and more languages, even some of the statically typed ones.

9.1.2. More power with metaprogramming

In the languages being developed today, we’re witnessing an increase in metaprogramming power. Ruby and Groovy offer runtime metaprogramming, as you saw in chapters 2, 3, 4, and 5. Clojure, the Lisp on the JVM, offers compile-time metaprogramming and lets you design expressive DSLs without one bit of runtime performance overhead. If you’re into DSLs, you need to master the metaprogramming techniques that your language of choice offers.

Statically typed languages like Haskell (see [10] in section 9.6) and OCaml (see [11] in section 9.6) have started to implement metaprogramming as part of the language infrastructure. Template Haskell is an extension to Haskell that adds compile-time metaprogramming facilities to the language. The traditional way to design DSLs in Haskell is to go for the embedded or the internal DSL implementation. Frequently there’s a mismatch between what the DSL developer wants to write and what Haskell lets you do with its syntax. Compile-time metaprogramming lets you write concrete syntax that can be converted to appropriate Haskell AST structures. It’s similar to what you can do with Lisp macros.

Developments in metaprogramming for many languages are a direct indication that DSLs are becoming mainstream. Next, we’re going to look at a feature that has the potential to replace most instances of XML as the carrier of data.

9.1.3. s-expressions instead of XML as the carrier

An expressive language like Clojure (or Lisp) provides you with s-expressions, which can model code as data. In today’s enterprise systems, you often see masses of XML being used as configuration data and touted as the DSL for expressive modeling. These XML structures are then parsed and processed using appropriate tools that generate executable artifacts for the application. The problem is that XML is painful to the eyes and has limited capability to express higher-order structures like conditionals. It serves as a poor alternative to s-expressions.

In one project I worked on, we were using XML for transporting entities as messages across various deployments. Consider the following XML snippet that models an Account object:

<account>
  <no>a-123</no>
  <name>
    <primary>John P.</primary>
    <secondary>Hughes R.</secondary>
  </name>
  <dateOfOpening>20101212</dateOfOpening>
  <status>active</status>
</account>

With XML as the format, we need to parse the message and transform it into an appropriate data structure. Why not use the s-expressions available in Clojure? The code becomes so much more expressive and less verbose at the same time:

(def account
 {no 123,
  name {primary "John P." secondary "Hughes R."},
  date-of-opening "20101212",
  status ::active })

The new snippet is more concise than the equivalent XML counterpart and is semantically much richer. It models data in Clojure that you can also execute. You don’t have to contrive additional machinery to parse the structure and transform it into a runtime artifact; it executes directly within the Clojure runtime. I call it executable XML. It’s a much better DSL than the XML version and is defined using only the features that your programming language offers. We’ll be seeing more of such data-as-code paradigm as DSL-based development matures.

Another trend that’s growing in popularity, mainly in functional languages, is using parser combinators. You saw in chapter 8 the power that parser combinators offer in DSL design. Let’s look at them again.

9.1.4. Parser combinators becoming more popular

You saw in chapter 8 how parser combinators allow you to design external DSLs even within the confines of a host language library. With functional programming becoming more popular, we’ll see a proliferation of parser combinator libraries. Gilad Bracha’s upcoming language Newspeak (see [4] in section 9.6) has a rich parser combinator library that can decouple grammar rules from the semantic model much better than what we have in Scala. Many existing languages like F# (see [5] in section 9.6), JavaScript (see [6] in section 9.6), and Scheme (see [7] in section 9.6) are also developing their own parser combinator libraries.

Parser combinators let you develop the syntax of your DSL in a declarative way, similar to writing EBNF rules. You can write EBNF-like declarative grammar rules in parser generators too, but when you use parser combinators, you remain within the scope of your host language and get to use all its other features. With support from the host language, you can decouple your semantic actions from the grammar and get a clean implementation of the DSL.

Let’s look at yet another stream of DSL development methodology. It’s on a higher level of abstraction than textual DSLs. I’m talking about DSL workbenches. You saw one example of this paradigm when we discussed DSL development with Xtext. DSL workbenches could very well bring about a fundamental change in the way we think about DSLs.

9.2. DSL workbenches

From a high-level view, a DSL design is, in some way, an exercise in building the most expressive API you can within the confines of your environment. In the case of internal DSLs, you’re limited to the host language that you use. With external DSLs, you design your own syntax, subject to the restrictions of your parser generator or combinator. In all these cases, we’re talking about textual DSLs; whatever interface you present to your user, it’s in the form of text-based structures. You can give very expressive APIs to your users, but if they’re implemented in a specific language, the user has to abide by the rules and regulations that the language runtime mandates.

Recently a school of thought has questioned this paradigm of text-based DSL development. Suppose I, as an expert in data analytics, want to embed Excel macros within the calculation engine of my weather forecasting system. To me, a spreadsheet seems to be the most intuitive way to express the logic that the macro encapsulates. In the text-only-based world of DSL design, there’s no way you can compose higher-order structures like a spreadsheet or a charting engine within the scope of your language.

Frameworks like Eclipse XText (see chapter 7) bring you a step in this direction. Instead of plain text, it stores the metamodel of the DSL, which can then be projected onto an Eclipse editor. The editor provides capabilities like syntax-highlighting and code completion. The higher the level of abstraction that such frameworks support, the easier it becomes for the end user to create, edit, and maintain her own DSL. A tool that supports end users in creating, editing, and maintaining DSLs is called a workbench.

9.2.1. What’s in a DSL workbench?

In chapter 7, you saw how you can generate workbenches for your DSL using Eclipse Xtext (see [1] in section 9.6). JetBrains Meta-Programming System (MPS) ([2] in section 9.6), and Intentional’s Domain Workbench ([3] in section 9.6) are similar tools in the same space. Instead of dealing with text-based programs, these tools use higher-order structures like the AST as the basic storage unit.

As a workbench user, you don’t have to write text-based programs; you’ll get a projectional editor, a special form of IDE, where you can manipulate your DSL structures. DSL workbenches usually offer seamless integration with tools like Microsoft Excel, which you can use to design your DSL syntax and semantics. The model that you build in Excel is stored in the workbench repository as metadata and corresponds to the higher-level abstractions of your DSL.

You can also generate code in specific languages from the metamodel that’s stored in the workbench’s repository. Wasn’t this supposed to be the domain user’s dream and the initial value proposition that we used to associate with DSLs? The domain workbench seems to be the ideal confluence of the domain experts and the programmers. Figure 9.4 shows how domain workbenches support the full lifecycle of a DSL implementation.

Figure 9.4. DSL workbenches support the full lifecycle of a DSL implementation. Domain experts work with higher-level structures like Microsoft Excel. The workbench stores metadata instead of program text. The metadata can be projected onto smart editors called projectional editors where you can edit, version, and manage it. The workbenches also have the facility to generate code to programming languages like Java.

The available DSL workbenches are based on the same principle of programming at a higher level of abstraction that we’ve been talking about in this book. There are differences, as far as the representations of abstractions are concerned. Some of the areas where these products vary are summarized in table 9.1.

Table 9.1. Feature variation in DSL workbenches

Feature	Differences between workbenches
Representation and definition of abstract syntax	The abstract syntax can be represented in terms of an abstract syntax tree or graph, and defined as a metamodel or a grammar form.
Metamodel composition	Many of the workbenches support an abstract syntax representation that’s a composition of several grammars or metamodels.
Transformation capabilities	Some workbenches, like Xtext, allow template-based code transformation, while MPS supports model-to-model transformation out of the box.
IDE support	Most of the workbenches offer powerful custom IDE support out of the box. They offer syntax highlighting, code completion, and context-sensitive help to the DSL writer.

A DSL workbench can definitely be beneficial to have in your bag of tricks. Let’s look at the advantages of using one.

9.2.2. The advantages of using a DSL workbench

Even though all workbenches vary in the degree to which they offer flexibility to the user with respect to the areas mentioned in the table, all DSL workbenches offer the following advantages:

• Separation of concerns between the interface of the DSL and the implementation.
• Direct interaction between the user and higher-level structures, structures that are on a higher level than those found in textual programming languages. The workbench approach to DSL development is far more appealing to nonprogramming domain users.
• A rich end-to-end environment for DSL-driven development.
• Easier composition of multiple DSLs.

When all’s said and done, DSL workbenches are still in the infancy stage, as far as adoption is concerned. The technology has promise and has been promoted for quite some time. Some concerns need to be addressed by the DSL workbench vendors before they can be positioned for mainstream adoption. The main concern is vendor lock-in with workbenches. The following attributes of a DSL workbench are the most important ones:

• Abstract representation schema
• Projectional editor
• Code generator

All of these are locked into the respective frameworks. There’s always a certain level of apprehension when you’re locked in a specific platform for modeling your DSL. Being locked in implies that your development team has to learn yet another specific tool set in order to implement a DSL as part of your project. Even so, workbenches are an interesting technology paradigm and we need to keep an eye on how they evolve.

Workbenches are one means to getting a complete DSL development environment. But besides workbenches, we’re also looking at enhanced tool support in IDEs that can make DSL development easier than what it is right now.

9.3. More tool support

As you saw in the earlier section, the primary tool support for designing DSLs comes in the form of DSL workbenches. But when you’re not using a workbench, how much support can you expect from the environment that you’re working with?

One obvious way of getting advanced tool support for writing DSLs is from your IDE. When you program in a general-purpose programming language using the support of an IDE, you get an editor that supports syntax highlighting, code completion, and many other editing features. Now imagine getting some of these features when you program using your DSL. Consider writing an internal DSL in Groovy for the financial brokerage system, where you want to highlight every currency code that the user enters. Or you want automatic code completion for some of the financial institutions supported by your system.

Many IDEs have started offering some sort of tooling that helps you with syntax highlighting and autocompletion, even without a full-blown DSL workbench. IDEs today are extensible using a plug-in-based architecture. You can plug in your own bits to define syntax highlighting, code completion, and many other things (see [8] and [9] in section 9.6).

The blog post Contraptions for Programming (see [8] in section 9.6) describes a plug-in for Eclipse-Groovy-based DSL development where you implement your own custom syntax highlighter. Eclipse-Groovy components provide an extension point in the form of an interface that you can implement to customize the list of keywords that you want to be syntax-highlighted. There’s similar custom Groovy DSL support for IntelliJ IDEA where the plug-in implements autocompletion for methods and properties see ([9] in section 9.6). Look at figure 9.5 for an overview of how you can introduce the syntax highlighter for your custom DSL as part of IDE plug-in architecture.

Figure 9.5. In an IDE, besides the core part, you can implement your own plugins. For your DSL, you can design a syntax-highlighter as a plugin and introduce it alongside the rest of the IDE.

So far we’ve talked about DSL development. Another important issue with today’s DSL-based environment is the disciplined evolution of DSL versions. I’ll give you a brief overview in the next section of how you can streamline the growth of a DSL so that multiple versions can coexist.

9.4. The mature evolution of a DSL

Many of us use DSLs in our application development. We use DSLs mainly to model the components of our system that tend to change frequently, like the configuration parameters and the business rules. One area that needs to mature further is the discipline that we follow to evolve a DSL in the face of such changes. You need to think about your DSL’s evolution strategy even before you come up with the first version to be deployed.

9.4.1. Versioning your DSL

Depending on how your DSL is going to be used, you need to have a versioning strategy. If your DSL is going to be used solely by a closed group of users working as a cohesive unit, you might decide not to follow a specific versioning strategy. Whenever you need to make a change to fix a bug or to introduce new requirements, you can roll out the newer version and replace the earlier one. A simple note that points out issues of backward incompatibilities will accompany the new version.

But what if multiple groups of users are going to use your DSL? Then you’ll have to plan for incremental versioning strategies. Not all your users will be interested in getting the new release, so you need to employ both of the following strategies:

• Your version management in the code base must be able to branch out to maintain multiple releases.
• You must create specific deployment scripts that can deploy multiple versions of your DSL.

Whatever strategy you use, make sure it addresses the following issues that frequently come up with respect to the evolution of any specific software module:

• Handling backward compatibility
• Catering to specific user needs that you can’t roll out for general use

Many of these concerns are also applicable to software deployment in general and aren’t specific to DSLs. In the following section, we’ll discuss some of the practices you can follow during your design phase that will address many of these versioning issues.

9.4.2. Best practices for a smoother evolution of DSL

Suppose you’re using a third-party DSL in your application that’s been deployed in multiple customer locations. You need to add more features in your application when you discover that the new version of the DSL has exactly what you want. But the new version isn’t backward compatible with the version that you’ve been using. What are you going to do?

Consider another scenario where your DSL models the business rules of securities trading that can vary with the stock exchange where it’s deployed. It so happens that some of these rules change only for the Tokyo Stock Exchange and you need to roll out a new version that’s specific to the Tokyo deployments. Yikes! The horror stories about trying to manage multiple versions simultaneously are legend.

Let’s look at some of the things that you can do upfront to mitigate these teething problems and keep them from haunting you through many sleepless nights.

Implicit context for Better Version Evolution

Consider this fluent API-based internal DSL snippet in Ruby that we discussed in section 4.2.1:

Account.create do

  number      "CL-BXT-23765"
  holders     "John Doe", "Phil McCay"
  address     "San Francisco"
  type        "client"
  email       "[email protected]"

end.save.and_then do |a|

  Registry.registNer(a)
  Mailer.new
        .to(a.email_address)
        .cc(a.email_address)
        .subject("New Account Creation")
        .body("Client account created for #{a.no}")
        .send
end

In this DSL, the account creation process uses the implicit context pattern of internal DSL design. This pattern makes the DSL easier to evolve when compared to the approach of making them fixed-position parameters to the create method. In the Account abstraction, you can add additional attributes without impacting existing clients.

Automatic transformation for backward compatibility

You can use this strategy to offer an automatic transformation of the older APIs to the newer ones, with appropriate defaults. Consider this snippet of a Scala DSL for defining a fixed income trade that we discussed in section 6.4.1:

val fixedIncomeTrade =
  200 discount_bonds IBM
    for_client NOMURA on NYSE at 72.ccy(USD)

Users were using this DSL happily. Trades were being made using the specific currency as mentioned in the DSL (USD in the snippet). This currency is called the trading currency. Eventually trades get settled through a settlement process, but our DSL assumed that the settlement of the trade was also being done in the same currency. As we all know, rules change, and one day users got a notification that a trade can be settled in a currency that’s different than the trading currency (called the settlement currency). Accordingly, the newer version of the DSL becomes:

val fixedIncomeTrade =
  200 discount_bonds IBM
    for_client NOMURA on NYSE at 72.ccy(USD)
settled in JPY

The question is what happens to the DSLs that were written using the earlier version of your engine? Those earlier DSLs are probably going to explode, because the underlying model won’t have a valid value for the settlement currency.

You can address this problem by defining an automatic transformation within the semantic model that sets up the default value of the settlement currency to the value of the trading currency. If you do this, users will have a migration path to follow and the earlier versions of the DSL will continue to run happily ever after.

A DSL facade can address a lot of versioning problems

Remember the DSL facade we talked about in section 5.2.1? A facade acts as a protector of your model APIs and helps you do manipulations with the syntax that you publish for your DSL users. When you need to make changes to the DSL syntax in future versions, you can localize your changes within the DSL facade without having any impact on the underlying model. This strategy works great if you need to roll out small syntax changes as part of newer versions of your DSL.

Follow the principles of well-designed abstractions

I’ve detailed principles of well-designed abstractions in appendix A. Read them, and then read them again. Every DSL you design must follow these principles if they’re going to evolve gracefully for your users. You need to version your DSL just like you need to version your APIs. The more rigid your APIs become, the more difficult it’ll be to make them evolve with newer versions.

Whatever option you choose, you need to make it possible to use multiple versions of the DSL in a single application. This area of DSL development is still evolving and needs more time to mature. You can help by carefully considering the future needs of your DSL users.

9.5. Summary

Here we are, at the concluding summary of the final chapter of the book. By now, you’ve been through all the aspects of how DSLs give you a better way to model your domain. In this chapter, we’ve looked at some of the future trends of DSL-based development. DSL workbenches promise more disciplined evolution of your DSL through an appropriate toolset that handles the complete lifecycle of your language. We’re seeing regular programming languages getting more expressive by the day, making them more suitable for use as the host language for your DSL. Whatever language you decide to use for developing your DSL, make sure you follow the discipline that helps you grow your DSL incrementally and iteratively.

In this book, I’ve discussed some of the JVM languages that have great features for designing DSLs. Besides standing out individually as powerful languages for DSL design, all of them nicely interoperate with Java using the common runtime of the JVM. This is a big plus, because as a user, you’re no longer restricted to using only one language for your DSL needs.

Besides these JVM languages, we’re seeing lots of other languages that are being used extensively for designing expressive DSLs. Haskell, the pure functional language, and Erlang, the language that supports concurrency-oriented programming, are the forerunners in this development. The software development community has realized that the only way to manage the complexity of domain modeling is to use languages that offer higher-order abstractions. DSL-driven development is one of the ways that make these abstractions into beautiful and reusable artifacts. A good DSL enhances productivity, makes code more maintainable and portable, and offers a friendly interface to the users. All of the unnecessary details are hidden away. A DSL is the way you should model a domain. We’ve already started to see the potential of DSLs in the real world of software development today.

 

 

9.6. References

1. Xtext User Guide. http://www.eclipse.org/Xtext/documentation/latest/xtext.html.
2. Meta Programming System. http://www.jetbrains.com/mps/.
3. Intentional Software. http://intentsoft.com/.
4. Newspeak. http://newspeaklanguage.org/.
5. Tolksdorf, Stephan. FParsec—A Parser Combinator Library for F#. http://www.quanttec.com/fparsec/.
6. Double, Chris. Javascript Parser Combinators. Bluish Coder. http://www.bluish-coder.co.nz/2007/10/javascript-parser-combinators.html.
7. Pretterhofer, Lorenz. Scheme Parser Combinators. A Lexical Mistake. http://alexicalmistake.com/2008/06/scheme-parser-combinators/.
8. Eisenberg, Andrew. Extending Groovy Eclipse for use with Domain-Specific Languages. Contraptions for programming. http://contraptionsforprogramming.blogspot.com/2009/12/extending-groovy-eclipse-for-use-with.html.
9. Pech, Vaclav. Custom Groovy DSL Support. JetBrains Zone. http://jetbrains.dzone.com/articles/custom-groovy-dsl-support.
10. Template Haskell. HaskellWiki. http://www.haskell.org/haskellwiki/Template_Haskell.
11. Meta OCaml. http://www.metaocaml.org/.