Chapter 6. Internal DSL design in Scala

This chapter covers

• Scala as a language
• Developing an internal DSL in Scala
• Composing multiple DSLs
• Using monadic structures

In the earlier chapters, we’ve been through the pros and cons of DSL-driven development. By now you must have realized that for the parts of your application that need to model business rules, a DSL can go a long way toward improving the communication path between the development team and the team of domain experts. In the last chapter, we discussed how you can use a few of the dynamic languages on the JVM as hosts for designing internal DSLs. In this chapter, we’ll look at the most promising candidate from among the statically typed ones, Scala.

Like all implementation discussions, this is going to be a programming-intensive chapter in which we’ll start with the suitability of Scala as a host for implementing internal DSLs, and then dive right into real-world DSL design. Figure 6.1 is a roadmap of our journey through this chapter.

Figure 6.1. Our roadmap through this chapter

Sections 6.1 and 6.2 will establish Scala as a host for internal DSLs. After that, we get into the details of implementing real-world use cases from our domain of securities trading back-office systems. You’ll see lots of idioms, best practices, and patterns in action in sections 6.3 through 6.6. In section 6.7, we’ll discuss how you can compose multiple DSLs to evolve larger ones. We’ll conclude the chapter with a discussion about how monads can make your DSLs concise, functional, and more expressive.

At the end of the chapter, you’ll have comprehensive knowledge of how to design DSLs using Scala as the host language. You’ll learn the idioms and best practices of how to model domain components and how to create easy-to-use and expressive language abstractions with them. Sounds exciting, doesn’t it? Let’s get started.

I used Scala 2.8 to produce the code listings and snippets. For those of you not familiar with the Scala syntax, there’s a Scala cheat sheet for you in appendix D at the end of the book.

6.1. Why Scala?

Scala offers a host of functional, as well as OO, abstraction mechanisms that help you design concise and expressive DSLs. A DSL can’t stand on its own; in case you missed it before, a DSL is layered as a facade on top of an implementation model. In this section, we’ll take a look at how Scala shapes up as a host language, both for designing the underlying model as well as the DSL layer on top of it. Table 6.1 shows some of the features of idiomatic Scala that you’ll use regularly to design your DSLs.

Table 6.1. Idiomatic Scala in DSL design

The confluence of the features listed in table 6.1 makes Scala one of the most powerful languages on the JVM for designing internal DSLs. But it’s still a new language that makes you think differently. Introducing a new language to a team is always a big change, both technologically and culturally. Your company might already have made a big commitment to the JVM as a platform, a number of your clients’ applications run on Java, and you have a whole lot of programmers trained in myriad Java frameworks. Are you going to forego all the benefits that you’ve accrued in order to embrace a new programming language? Fortunately, with Scala, you can afford to take an incremental step toward this transition. The next section discusses how.

6.2. Your first step toward a Scala DSL

The fact that Scala makes a good host for an internal DSL isn’t enough to convince your manager to introduce it as one of the technologies in your development ecosystem. Any new technology needs to be introduced gradually or the risk of introducing chaos increases. Typically you can roll out this adoption slowly by using it in a noncritical line of business.

With Scala, the big plus is that it is on the JVM and interoperates with Java. You can preserve all your Java investments and still try to adopt Scala in your enterprise. You can get into using Scala DSLs in quite a few ways; you can even design a few of your own while keeping your base Java abstractions. Figure 6.2 shows some of the strategies you can use with your own development team.

Figure 6.2. You don’t need to start doing Scala in production code from day one. These are some of the baby steps that you can start with, in no specific order, during the lifetime of your project.

As you can see from this figure, your mainstream delivery can continue in Java, while some members of your team get inducted into Scala by working in auxiliary activities of your project. Let’s look at each of these ways of introducing Scala in detail in the following sections.

6.2.1. Testing Java objects with a Scala DSL

Testing is one of those activities that forms the core of your development practice. At the same time, it gives you plenty of flexibility in choice of technology and frameworks. Test suites need to be treated as artifacts of equal importance to your code base. You’ll see lots of development going on in the industry that attempts to make test suites more expressive and exhaustive.

DSLs have become an integral part of testing frameworks. Choose a Scala DSL-based testing framework and you can begin your journey to learning DSL designs in Scala today. ScalaTest (see [8] in section 6.10) is one such framework that lets you write DSLs to test Java as well as Scala classes. You don’t need to be able to write Scala classes from the get go. You can reuse all your Java classes with these testing frameworks and get the feel of working in a DSL-based environment.

6.2.2. Scala DSL as a wrapper for Java objects

As we’ve discussed many times in this book, Scala integrates quite seamlessly with Java. You can dress up Java objects with Scala wrappers to make them more smart and expressive. In case you’re not convinced, go back to section 3.2.2 where we implemented lots of smartness on top of Java objects using Scala power. This approach can be a prudent way to learn how to make DSLs on top of your Java object model using Scala.

6.2.3. Modeling noncritical functionality as a Scala DSL

In a large application, there are often areas that aren’t so critical. You can negotiate with the client regarding their deliveries. You can select some of these as hotspots for learning to design Scala DSLs. If you don’t want to get away from your main Java compile mode, you can even script your Scala DSL and run it using the ScriptEngine that comes with Java 6.

Let’s get down to the next level of detail and discuss how you can use these features in Scala to build domain models that help you make good DSLs.

Code Assistance

In all of the following sections that have rich code snippets, I’ve included a sidebar that contains the prerequisites of the language features that you need to know in order to appreciate the implementation details. Feel free to navigate to the appropriate language cheat sheet that’s in the appendixes as you encounter this code.

We’ll continue using the same domain of financial brokerage solutions and build examples of how these features combine to form the wholeness of a complete DSL ready to run. It is going to be an exciting journey.

6.3. Let’s DSL in Scala!

You have enough of a background now to get into what we’re here for. We’ll study real-life use cases from the domain of securities trading and see how they translate to expressive DSLs using Scala as the implementation language.

We’ll be examining use cases similar to the ones I selected for the discussions about designing DSLs in Ruby and Groovy. That way, you’ll see how you need to think differently, even with the same problem domain, when you’re designing DSLs in both statically typed and dynamic languages. But first let’s look at some of the features that Scala offers that make the syntax of your DSL expressive to users.

6.3.1. Expressive syntax on the surface

When you talk about the syntax of a language, there’s always a fine line between expressiveness and verbosity. A syntax that’s expressive to a nonprogramming user might seem extremely verbose to a programmer. We discussed this issue when you designed an interpreter in Ruby for the trading DSL in section 5.2.3. Remember how Bob complained about the nonessential syntactic complexities that Java forced on us in the order-processing DSL that we designed in section 2.1.2? Despite being statically typed like Java, Scala offers a more concise surface syntax, which makes the DSL less noisy. Consider the following listing, which shows typical Scala code that adds a ClientAccount to an already existing list of accounts.

Listing 6.1. Expressive and concise syntax in Scala

Even if you’re not familiar with Scala, this listing makes perfect sense. Table 6.2 shows some of the features that make this snippet so concise.

Table 6.2. Features that make Scala syntax concise, with reference to listing 6.1

Conciseness in Scala	Impact on your DSL design
Semicolon inference	Unlike in Java, you don’t need to put in semincolons as delimiters between Scala statements. Less noise is the immediate result when you design DSLs in Scala.
Named and default arguments	We instantiate a ClientAccount class using named arguments. The class declaration is implemented as a case class, which gives you a lightweight syntax for construction. The class declaration can also contain default values for arguments that don’t need to be repeated during instantiation. Using these arguments provides valuable support toward improved readability of your DSL script.
Type inference	When you construct a list with the accounts, you don’t need to specify the type of the resulting list . The compiler infers that for you.
Operators as methods	The list accounts is augmented with another ClientAccount using the operator syntax :: . This is actually another way to do a method dispatch on the List instance accounts.::(ClientAccount(no = "acc-345", name = "Hugh P."). Note how the operator syntax and the optional dot (.) for method invocation make the code fragments much more readable to the user.
Optional parentheses	We drop the first account from the list using the drop method on List . The parentheses are optional here, making the code fragment more aligned to natural flow.

The items that we discussed in this section are purely syntactical issues on the surface. Other features also contribute to readable syntax in Scala, including powerful literal syntax for collections, use of closures as control abstractions, and advanced features like implicit parameters. We’re going to look at them separately later in this chapter when we discuss the roles that each of them play in designing an internal DSL.

Let’s start by designing some of the domain abstractions that form the base on which you’ll build your DSLs. We’ll use our trusty domain of securities trading and the same abstractions that I relied on in earlier chapters using other languages. Not only does this approach help you connect to domain concepts that you’ve already learned, it also makes a good comparative study of implementation idioms across all the languages that you’ve seen so far.

6.3.2. Creating domain abstractions

When you design a DSL in Scala, it’s mostly an object model that serves as the base abstraction layer. You implement specializations of various model components using subtyping, and form larger abstractions by composing with compatible mixins from the solution domain. For the actions within your model, you create functional abstractions, then compose those using combinators. Figure 6.3 explains the ways you can achieve extensibility in Scala abstractions with the dual power of OO and functional capabilities.

Figure 6.3. With Scala you can use the dual power of OO and functional programming to evolve your domain model. Using the OO features of Scala, you can abstract over types and values, specialize a component through subtyping, and do composition using mixins. You can also use the functional features of Scala through its higher-order functions, closures, and combinators. Finally, you can compose all of this using modules and get your final abstraction.

To build your implementation of the trading DSL, let’s look at some of the base abstractions from the problem domain.

The Instrument

Listing 6.2 is an abstraction of an Instrument that you trade in a stock exchange. It starts with the general interface for an Instrument, then specializes into Equity and various forms of FixedIncome securities. (If you need an introduction to the various instrument types, read the sidebar accompanying section 4.3.2.)

Listing 6.2. Instrument model in Scala

The domain vocabulary is explicit in the implementation and we don’t have much noise from accidental complexities polluting the essence of the domain model. (For more about accidental complexity, see appendix A.) Because this is the first of the domain models that we’ll carve out in this chapter, let’s look at some of the Scala features that make this model expressive yet concise.

• Singleton objects, implemented as specializations of the Currency class , which are instantiated exactly once. This is the Scala way of implementing the Singleton pattern (see [3] in section 6.10). It avoids all the evils of statics in Java.
• Extensible object hierarchies through traits that can be extended through inheritance .
• Simplified constructor invocations for case classes.

Let’s look at a couple more abstractions before we start building some DSL scripts out of them.

Account and Trade

The following listing is the Account model in Scala. Account is the domain entity against which clients and brokers trade securities.

Listing 6.3. Account model in Scala

Now that we have the Account and Instrument models ready, we can define the base abstraction for security trade.

Listing 6.4. Trade model in Scala

We define two types of trades, depending on the class of instrument being traded. As you’ll see later, the two types of trades have different characteristics with respect to how their cash values are calculated. (For what I mean by the cash value of trade, see the sidebar accompanying section 4.2.2.) Also note that we override the instrument method in listing 6.4 to reflect the correct type of security that the trade deals with.

Well, that was quite a bit of coding to do only to set up the context for taking your first shot at writing a trade creation DSL. You’ll get your chance to do that in the next section, where there’s also a discussion of some of the Scala features that help you build it.

6.4. Building a DSL that creates trades

I’m a firm believer in looking at concrete things first and exploring how they evolve. Without going into any of the implementation specifics, here’s how our trading DSL creates new trades for you:

val fixedIncomeTrade =
  200 discount_bonds IBM
    for_client NOMURA on NYSE at 72.ccy(USD)
val equityTrade =
  200 equities GOOGLE
    for_client NOMURA on TKY at 10000.ccy(JPY)

The first definition, fixedIncomeTrade, creates an instance of FixedIncomeTrade to buy 200 discount bonds (DiscountBond) of IBM for client account NOMURA at 72 USD per unit traded on the New York Stock Exchange.

The second definition, equityTrade, creates an instance of EquityTrade for sale of 200 equities of GOOGLE for client account NOMURA at 10000 JPY per unit traded on the Tokyo Stock Exchange.

Now let’s look at the regular API version of a trade-creation process that uses the constructor of one of the concrete classes. The following listing shows the concrete implementation of FixedIncomeTrade, followed by a sample instantiation.

Listing 6.5. FixedIncomeTrade implementation and instantiation

The difference between the DSL and the more typical API is obvious. The DSL version looks more natural and readable to a domain user, but the API has the feel of a program fragment. You’ve got to take care of quite a few syntactic nuances in the second version: commas as argument separators, usage of the class name for instantiation, and so on. As you’ll see later, implementing a readable DSL version also imposes quite a few constraints as far as sequencing operations are concerned. You can opt for flexible sequencing using the Builder pattern (see [3] in section 6.10), but then you have to deal with two additional issues: the mutability of Builder objects and the finishing problem (see [4] in section 6.10).

Now let’s dig into the implementation aspects of the DSL script that I showed you at the beginning of this section.

6.4.1. Implementation details

Before looking at the details, take a hard look at the DSL script at the beginning of section 6.4 and the normal constructor invocation in listing 6.5. The most salient difference between them that you’ll notice is the absence of any (well, almost any) additional nonessential syntactic structure in the DSL version. As I mentioned earlier, syntaxes like the dot operator for method invocation or parentheses for method arguments are optional in Scala. But you still need to wire them up with enough flexibility so that everything makes sense in the final script. What do you think is the secret sauce for this wiring?

Implicit Conversion

It’s Scala implicits! Let’s start with the creation of the FixedIncomeTrade:

val fixedIncomeTrade =
  200 discount_bonds IBM
    for_client NOMURA on NYSE at 72.ccy(USD)

If we take the sugar out of this statement and put back the dots and parentheses to make it more canonical, it becomes:

val fixedIncomeTrade =
  200.discount_bonds(IBM)
     .for_client(NOMURA)
     .on(NYSE)
     .at(72.ccy(USD))

With all the method invocations and arguments decorated with their honored symbols, it looks similar to the Builder pattern that we used in section 2.1.2 to build our order-processing DSL in Java. For the current implementation, you can say we’re using an implicit form of the Builder pattern. Yes, we’re literally using the implicit conversion feature of Scala to do the conversions so that each individual piece gets wired up in the right sequence to render the final form of the DSL.

Let’s look at how 200 discount_bonds IBM makes sense. When you understand the mechanics that build this stuff, you’ll be able to figure out how the rest of the things fall in place by looking at the complete code. Look at the following code snippet:

type Quantity = Int
class InstrumentHelper(qty: Quantity) {
  def discount_bonds(db: DiscountBond) = (db, qty)
}

implicit def Int2InstrumentHelper(qty: Quantity) =
  new InstrumentHelper(qty)

We define a class InstrumentHelper that takes an Int and defines a method discount_bonds. The method takes an instance of DiscountBond and returns a Tuple2 of the bond and the quantity. Then we define an implicit conversion from Int to the class InstrumentHelper. This conversion converts an Int implicitly to an instance of InstrumentHelper on which we can invoke the method discount_bonds. Because Scala has optional dots and parentheses, you can use the infix form 200 discount_bonds IBM to make it look more natural.

After you define this conversion, the Scala compiler takes care of the rest at use-site by adding the necessary call semantics to your script. This same mechanism works for the rest of the script and ultimately results in a method that can generate an instance of a FixedIncomeTrade with all necessary arguments. We’ll look at the complete code and see some of the idioms that you need to follow to use implicit conversions. But first, look at figure 6.4, which traces the execution of the script to generate the trade.

Figure 6.4. Sequence of implicit conversions that leads to the construction of the FixedIncomeTrade instance. Read the figure from left to right and follow the arrows for implicit conversions and the subsequent creation of helper objects.

To understand the figure, you need to look at the complete source code of the object that does this implicit magic behind your API.

A Bag of Implicit Conversions

If you look closely at figure 6.4, you’ll realize that it’s really a seesaw of implicit conversions that plays the role of an implicit builder. These conversions evolve the final FixedIncomeTrade object. The code in the following listing defines the helper functions that do the conversions.

Listing 6.6. TradeImplicits defines the conversion functions

package dsl

import api._
object TradeImplicits {

  type Quantity = Int
  type WithInstrumentQuantity = (Instrument, Quantity)
  type WithAccountInstrumentQuantity =
    (Account, Instrument, Quantity)
  type WithMktAccountInstrumentQuantity =
    (Market, Account, Instrument, Quantity)
  type Money = (Int, Currency)

  class InstrumentHelper(qty: Quantity) {
    def discount_bonds(db: DiscountBond) = (db, qty)
  }

  class AccountHelper(wiq: WithInstrumentQuantity) {
    def for_client(ca: ClientAccount) = (ca, wiq._1, wiq._2)
  }

  class MarketHelper(waiq: WithAccountInstrumentQuantity) {
    def on(mk: Market) = (mk, waiq._1, waiq._2, waiq._3)
  }

  class RichInt(v: Int) {
    def ccy(c: Currency) = (v, c)
  }

  class PriceHelper(wmaiq: WithMktAccountInstrumentQuantity) {
    def at(c: Money) = (c, wmaiq._1, wmaiq._2, wmaiq._3, wmaiq._4)
  }
  //..
}

The next listing continues with the same object TradeImplicits and defines the conversion functions shown in listing 6.6 as implicit definitions in Scala.

Listing 6.7. The implicit definitions in TradeImplicits

object TradeImplicits {

  // .. continued from listing 6.6

  implicit def quantity2InstrumentHelper(qty: Quantity) =
    new InstrumentHelper(qty)
  implicit def withAccount(wiq: WithInstrumentQuantity) =
    new AccountHelper(wiq)
  implicit def withMarket(waiq: WithAccountInstrumentQuantity) =
    new MarketHelper(waiq)
  implicit def withPrice(wmaiq: WithMktAccountInstrumentQuantity) =
    new PriceHelper(wmaiq)
  implicit def int2RichInt(v: Int) = new RichInt(v)

  import Util._
  implicit def Tuple2Trade(
    t: (Money, Market, Account, Instrument, Quantity)) =
    {t match {
      case ((money, mkt, account, ins: DiscountBond, qty)) =>

      FixedIncomeTradeImpl(
        tradingAccount = account,
        instrument = ins,
        currency = money._2,
        tradeDate = TODAY,
        market = mkt,
        quantity = qty,
        unitPrice = money._1)
    }
  }
}

The object TradeImplicits is in a package named dsl, but all the domain model abstractions are in a package named api. This isn’t as unnecessary as it might seem. Remember when we talked about the underlying domain model that forms the base on which you build the DSL facade? In this example, all domain model abstractions are in the package api, while the linguistic layer is kept in dsl. Also, you need to keep these two layers decoupled so you can have multiple DSLs from the same domain model. Always maintain this convention when you’re designing your DSLs.

In Scala, implicits give you the power of open classes, similar to monkey patching in Ruby or ExpandoMetaClass in Groovy. At the same time, Scala gives you a way to control the visibility of the class you open up for modification. Import the specific module only within the lexical scope that uses these additional methods and the compiler will take care of the rest. The global namespace isn’t polluted, like it is in the Ruby counterpart.

Implicits and lexical scope

Using implicits, we added a method named ccy to Int through an implicit conversion to the RichInt class. If we keep this implicit conversion at the global namespace, all threads will be able to see this change. We already discussed the obvious drawbacks of this arrangement when we talked about Ruby monkey patching earlier. Make this your golden rule: implicits must be scoped appropriately. In this case, do an explicit import TradeImplicits._ and make the implicit conversion available only to your lexical scope, without impacting any other thread of execution.

Still, when all’s said and done, implicit conversions aren’t visible explicitly within your code and might give off a magical vibe when you’re debugging. To help demystify things, Scala has compiler switches that let you check implicit conversions as a post-compilation debugging tool (see [2] in section 6.10).

This example is the first Scala DSL that you’ve written. Aren’t you excited about the expressiveness it has? If you’re not comfortable yet with the ins and outs of the implementation of the DSL, go back and re-examine figure 6.4. Make sure your understanding of the code base flows in the same path as the progression shown in the figure.

OK. Now it’s time to settle down after almost being swept away by the wave of Scala excitement. Let’s compose ourselves and think about some of the key issues that you need to be aware of when you’re deciding in favor of creating a Scala DSL.

6.4.2. Variations in DSL implementation patterns

If you look carefully at the code that we developed as the layer that’s on top of the domain model for building the DSL, you’ll notice a pattern. This pattern is also explicit in the diagram that I presented in figure 6.4. Move from left to right in the diagram as the DSL script gets interpreted. You’ll notice how we build an n-tuple cumulatively through a successive application of implicit conversions in Scala. This phenomenon is effectively the Builder pattern that I mentioned in section 6.4.1. But unlike the traditional builder approach in which we have a separate mutable abstraction that builds the entity, here we’re using an immutable variant of the same pattern. In the imperative version of the Builder pattern, the builder object is updated with successive method invocations, each of which returns an instance of the self. In this case, the methods don’t belong to the same class and the implicit conversion of Scala acts as the glue that binds the invocations together. One invocation generates a tuple; that tuple gets converted implicitly to the next stage of the pipeline, which takes the earlier tuple and generates the next one.

You could have just as easily used the other pattern. Would it have made any difference? The traditional Builder pattern gives you the convenience of flexible sequencing of the method calls. The problem is that you have to invoke the finishing method to complete the build process (see [4] in section 6.10). In the current implementation, the sequence is fixed in the DSL and the compiler will complain if you finish the sequence prematurely without building the complete trade. As usual, it’s a trade-off, like many other design decisions.

The traditional Builder pattern uses a mutable builder object. You invoke method chaining through fluent interfaces that mutate the builder object. In the form of the Builder pattern that you just saw, which evolves through implicit conversions, every object is immutable, which is one of the recommended idioms in abstraction design.

Before we conclude this section, let’s look at the key aspects of some of the Scala features that you learned in order to build the DSL facade on top of your domain model abstractions. Table 6.3 contains a summary of this information.

Table 6.3. Scala features checklist for trade-creation DSL

You’ve completed the DSL for creating trade objects. In the next section, you’ll build more DSLs for business rules, each of which can be verified by the domain experts. Remember that the main value-add of DSL-based development is to foster better communication with the domain experts and help them verify the business rules that you’ve implemented. Before you develop the DSL, you need to have some more domain abstractions that serve as the underlying implementation model.

6.5. Modeling business rules with a DSL

Business rules are the hotspots for using DSLs. They form the configurable sections of your domain model and are the most important things that you need to verify through your domain experts. It’s an added benefit if your DSL is friendly enough for your domain experts (like our friend Bob) to be able to write a few tests around them. For our DSL, the business rule that we need to model is that the tax and fees for a trade must be calculated. See table 6.4 for how that takes place.

Table 6.4. Business rule to model with DSL: calculate tax and fees for a trade

The DSL that you’ll design needs to be readable enough for Bob, our domain expert, to understand the rules, check for the comprehensiveness of implementation, and certify its correctness. What do you need to do first? You guessed it! You need to create the domain model abstractions for tax and fee components before you try to add the DSL layer on top of them.

I suspect that some of my more astute readers are getting a bit impatient at the prospect of having to sit through another session of domain modeling before getting a peek at the next serving of DSL implementation. Let’s do something more interesting. Let’s digress and build a small DSL that implements a business rule that’s based on the domain model that we’ve already implemented. This exercise will both perk you up and demonstrate one of the functional features of Scala that’s used extensively to make a better implementation of one of the most commonly used OO design patterns.

6.5.1. Pattern matching as an extensible Visitor

Besides offering a simplified constructor invocation syntax, case classes in Scala use pattern matching over deconstructed objects, a feature typically used by algebraic data types in functional languages like Haskell. (For more information about algebraic data types, go to http://en.wikipedia.org/wiki/Algebraic_data_type. For more details about how pattern matching works in Scala, see [2] in section 6.10.) The reason for using pattern matching over case classes is to implement a generic and extensible Visitor pattern (see [3] in section 6.10).

In DSL design, you can use the same pattern to make your domain rules more explicit to users. Although with a typical OO implementation such rules tend to be buried within object hierarchies, you can use this functional paradigm over your case classes to achieve a better level of expressiveness and extensibility. For more details about how pattern matching over case classes in Scala leads to more extensible solutions compared to a traditional OO Visitor implementation, see [5] in section 6.10.

Consider another business rule that we’ll implement as a DSL in our application: Increase the credit limit of all client accounts that were open before today by 10%.

Listing 6.3 is the Account abstraction of our domain model with two concrete implementations for ClientAccount and BrokerAccount. (Remember that we discussed client accounts in a sidebar in section 3.2.2. A broker account is an account that the broker opens with the stock trading organization.) The implementation of the proposed rule needs to abstract over all client accounts that are present in the system and that are affected by this change in the credit limit. Let’s look at the Scala snippet that implements this rule in the function raiseCreditLimits.

Note how the rule is explicitly published through pattern matching over case classes. Under the hood, the case statements are modeled as partial functions, which are defined only for the values mentioned in the case clauses. Pattern matching makes modeling the domain rule easy, because we care only about ClientAccount instances in the current context. The underscore (_) in the second case clause is a don’t-care that ignores other types of accounts. Refer to [2] in section 6.10 for more details about pattern matching and partial functions in Scala.

Why is this a DSL? It expresses a domain rule explicitly enough for a domain expert to understand. It’s implemented over a small surface area, so that the domain person doesn’t have to navigate through piles of code to explore the semantics of the rule. Finally, it focuses only on the significant attributes that the rule specifies, blurring the nonessential parts within a don’t-care clause.

Now that you have an early taste of yet another DSL fragment that models a sample business rule for our solution, let’s get into the domain model of tax and fee that we promised earlier. The next section is going to be exciting. You’ll learn lots of new modeling techniques that Scala offers. So grab another cup of coffee and let’s get going.

6.5.2. Enriching the domain model

We built Trade, Account, and Instrument abstractions earlier. Those were the basic abstractions from the problem domain. Now let’s consider the tax and fee components that need to interact with the Trade component to calculate the cash value of the trade.

When we talk about tax and fee, we need a separate abstraction that’s responsible for their calculation. Calculating tax and fee is one of the primary business rules of the model that’ll vary with the country and the stock exchange where you’ll deploy your solution. And as you must’ve figured out by now, for business rules that can vary, a DSL makes your life easier by making the rules explicit, expressive, and easier to maintain.

Figure 6.5 shows the overall component model of how the tax and fee abstractions interact with the Trade component in our solution model.

Figure 6.5. Tax fee component model for the trading solution. The class diagram shows the static relationship between the TaxFeeCalculationComponent and the collaborating abstractions.

Note that all of abstractions depicted in figure 6.5 are modeled as Scala traits. As such, they can be wired together flexibly, and composed with suitable implementations to generate the appropriate concrete object during runtime. Let’s look at the TaxFeeCalculator and TaxFeeCalculationComponent in the following listing.

Listing 6.8. Tax and fee calculation components in Scala

Let’s look into this code listing and try to understand how the entire component model gets wired up. Table 6.5 has the details.

Table 6.5. Dissecting a Scala DSL implementation model

Abstraction	Role in the DSL implementation
TaxFee	This abstraction is the value object (see [6] in section 6.10) that corresponds to the individual tax and fee types. The various tax/fee types are modeled as singleton objects in Scala . Note: As value objects, all individual tax/fee types are immutable.
TaxFeeCalculator	Abstraction that calculates all the taxes and fees applicable to a trade .
TaxFeeCalculationComponent	This is the overarching abstraction that wires up a couple of other abstractions and forms the core that does the actual calculation of taxes and fees for a trade. TaxFeeCalculationComponent collaborates with TaxFeeRulesComponent through a self-type annotation , and TaxFeeCalculator through an abstract val . Design benefits: The abstraction is decoupled from the implementation. You’re free to provide implementations for both of the collaborating abstractions of TaxFeeCalculationComponent. Implementation can be deferred until you create concrete instances of TaxFeeCalculationComponent.

It looks like we’ve achieved a lot of wiring without much coding. Limited coding is the power that Scala brings to you; you can program at a higher level of abstraction. In the next section, we’re going to complete both the implementation of TaxFeeRulesComponent and a DSL for defining domain rules for calculating tax and fee.

6.5.3. Calculating tax and fee business rules in a DSL

Let’s start the domain model of the rules component with a trait that publishes the main contracts for tax and fee calculation. For brevity, we’ll consider only a simplified view of the world here; in reality, things are way more detailed and complex.

The first method, forTrade , gives a list of TaxFee objects that are applicable to the specific trade. The second method, calculatedAs , does the calculation for a specific TaxFee valid for the particular trade.

Now let’s look at the TaxFeeRulesComponent, which, along with building the DSL for calculating the tax and fee, provides a concrete implementation of TaxFeeRules. This component is shown in the following listing.

Listing 6.9. DSL for tax and fee calculation business rules

TaxFeeRulesComponent abstracts over TaxFeeRules and provides an implementation of it. You can supply your own implementation if you want, but TaxFeeRulesComponent is still an abstract component because it contains an abstract declaration of taxFeeRules. We’ll provide all the concrete implementations when we compose our components together, building a concrete TradingService. But first let’s take a detailed look at the implementation shown in the listing to see how the DSL gets the tax and fee types, then goes on to calculate the tax and fee amount.

Getting the list of applicable tax and fee heads

Let’s look first at the implementation of the DSL in TaxFeeRulesImpl. The method forTrade is a single-line method, which is a functional composition using Scala combinators. As you read in appendix A, combinators are a great way to compose higher-order functions. (If you haven’t read appendix A, you’re missing out.)

Combinators play a role in making DSLs expressive. They shine as one of the most useful areas of functional programming. Scala offers you the power of functional programming; feel free to use combinator-based language construction whenever you think it’s appropriate. The business rule for finding the set of taxes and fees for a trade stated in English is as follows:

• “Get the Hong Kong-specific list for trades executed on the Hong Kong market OR Get the Singapore-specific list for trades executed on the Singapore market OR Get the most generic list valid for all other markets.”

val onlyTrue: PartialFunction[Boolean, Int] = {
  case true => 100
}

scala> onlyTrue isDefinedAt(true)
res1: Boolean = true
scala> onlyTrue isDefinedAt(false)
res2: Boolean = false

Now read the single statement in forTrade , which implements this rule. You’ll see an exact correspondence to the most natural way of expressing the rule and you get it all within a small surface area of the API. We used the combinator orElse, which allows you to compose partial functions in Scala and select the first one that defines it.

In listing 6.9, the composed abstraction in method forTrade returns the generic list of TaxFee objects only if the market is neither Hong Kong nor Singapore. When you understand how forTrade works and how you can compose partial functions in Scala, you’ll know how the specific higher-order functions forHKG , forSGP , and forAll work.

Calculating the tax and fee

It’s now time to look at how the taxes and fees are calculated. This calculation is the second part of the business rule that we’re addressing in our DSL. Look at the method calculatedAs in listing 6.9. Can you figure out what rule it implements?

Once again, we see Scala pattern matching making domain rules explicit. For each case clause, the return value is once again sugared with the magic of implicits that adds a method percent_of to the class Double. The result is the infix notation that you see in listing 6.9. And here’s the TaxFeeImplicits object that you need to import to bring all implicit conversions to the scope of your DSL:

package api

object TaxFeeImplicits {
  class TaxHelper(factor: Double) {
    def percent_of(c: BigDecimal) = factor * c.doubleValue / 100
  }

  implicit def Double2TaxHelper(d: Double) = new TaxHelper(d)
}

After you import the TaxFeeImplicits object, you get the domain-friendly syntax in the method calculatedAs, which your business user will like a lot.

A DSL and an API: what’s the difference?

In section 6.5, you’ve learned how to make DSL scripts for creating domain entities on top of an underlying implementation model. You learned how to build DSLs for your business rules. I described some of the techniques that Scala gives you to implement expressive APIs over an OO domain model. In both of the implementations we worked through, I went a bit overboard and tried to make our little language more expressive using the open classes that implicit conversions offer. But even without the added sweetness of implicits, you can make your domain model implement sufficiently expressive APIs using the combination of OO and functional programming features.

This fact brings to mind a question that has surely crossed your mind as well: what’s the difference between an internal DSL and an API? Frankly speaking, there’s not much of a difference. An expressive API that makes the domain semantics explicit to its users without the burden of additional nonessential complexities is an internal DSL. In all the code snippets that I’ve branded as DSLs, the driving force is domain expressiveness for the user. The implementer of the DSL needs to maintain the code base, the domain expert needs to be able to understand the semantics; you can achieve both of these without going overboard. But you can do that only if you’re using a language that enables you to build higher-order abstractions and compose them together. Maybe it’s time now to take another look at the virtues of well-designed abstractions that I point out in appendix A.

As mentioned earlier, all the components I’ve described so far in figure 6.5 are abstract, in the sense that we’ve designed them as traits in Scala. You’ve yet to witness the real power of composing traits to form concrete instantiable domain abstractions. Let’s compose the trade abstractions in the next section and build some concrete trading services. After we have the services, they’ll serve as the base for developing the linguistic abstractions of our DSL.

6.6. Stitching ’em all together

Now that you’ve built a DSL that addresses the business rule for calculating the tax and fee, let’s build some new abstractions that’ll be the spice for the next serving of DSL.

In this section, you’ll learn how to compose traits through mixin-based inheritance in Scala. You’ll also see another form of abstraction that Scala supports: abstracting over types. When you have more options to use when you’re composing your abstractions, you can make your domain model more malleable, and your DSL syntax can evolve more easily out of it.

6.6.1. More abstraction with traits and types

When you design a domain model, one of the abstractions that you publish to your end clients is a domain service. A domain service uses entities and value objects (see [6] in section 6.10) to deliver the contracts they expose to clients. Let’s look at a typical domain service, called TradingService, in the following listing, much simplified compared to a real-world use case.

Listing 6.10. Base class for TradingService in Scala

Let’s make a quick run down of the service contracts and some of the new Scala features that they use in table 6.6.

Table 6.6. Dissecting the domain service TradingService in listing 6.10

Feature	Description
Power of mixins—composing with existing abstractions	TradingService mixes in with two of our earlier components TaxFeeCalculationComponent and TaxFeeRulesComponent . Note: With mixins we get inheritance of the interface as well as optional implementations. This is multiple inheritance, done right.
Abstraction over the type of trade	The trait TradingService abstracts over the trade type . This kind of maneuver is intuitive because we need to specialize the trading service, depending on the type of trade it handles. But there’s an upper bound on the constraint of our base class for security trade, Trade. When do you concretize T? When we concretize TradingService later, we’ll supply an implementation for the abstract trade type T
The core logic of the tax fee calculation is totalTaxFee	The service defines a concrete method totalTaxFee that sums over the component tax and fee items using the foldLeft combinator. For more details about how foldLeft works with the placeholder syntax (_) of Scala, read appendix D at the end of the book. Tip: Always prefer combinators to explicit recursion or iteration.
Abstract method for deferred implementation in subclasses	cashValue is an abstract method that we’ll define in subtypes, because the actual logic depends on the type of trade that the service handles.

Note that we haven’t yet made any abstraction concrete; they’re still abstract, with traits having abstract types that we’ll define in the next section. Scala as a language offers a variety of options to design your abstractions. Choose the ones that best fit the problem at hand and make your design aligned to the idioms of well-designed abstractions that I discuss in appendix A.

6.6.2. Making domain components concrete

EquityTradingService provides the trading service for equity trades. It’s a concrete component that needs to be instantiated once for all services that it renders. You model it using the singleton object notation of Scala (see [2] in section 6.10) in the following listing.

Listing 6.11. A concrete trading service for equity trades

This notation looks pretty straightforward, doesn’t it? It contains the following elements:

• A concrete type EquityTrade for the abstract trade type we defined in the base class
• Concrete implementations for the values we left as abstract in the traits that we mixed in
• A definition of how to compute the cashValue of an equity trade

Similar to EquityTradingService, we also implement another concrete trading service FixedIncomeTradingService, the counterpart for the FixedIncomeTrade class, in the following listing.

Listing 6.12. Trading service for fixed income trades in Scala

Note the additional component that we mix in with the core abstraction, Accrued-InterestCalculationComponent , which computes accrued interest for the trade. Accrued interest is something typical to fixed-income instruments and also forms an integral part of the cash value calculation for fixed income trades. I’m sure it’s obvious as well from how we define the FixedIncomeTradingService abstraction.

In this section, we defined service abstractions for our domain. Then we wired them up with the components that we built earlier to construct concrete Scala modules that you can directly use within your DSL.

The real power that Scala offers you in this exercise is the ability to defer committing to a specific implementation until the last stage. Abstract vals, abstract types, and self-type annotations are the three main pillars that helped you achieve this. Add to them the flexibility of composing abstractions through mixin-based inheritance and you have the complete recipe for designing scalable components.

What you’ve done so far is construct a DSL from a set of underlying domain model components. In Scala, I defined the DSL layer as a set of abstractions that evolve based on the requirements of the user. In this way, you end up with hierarchies of abstractions that model the various use cases of our trading system. Now consider what happens when some of the market rules change and you need to integrate the new rules with a set of existing abstractions. Here we’re talking about composing the existing DSL with some new ones. In the next section, I’ll show you how you can do this using Scala’s type system.

6.7. Composing DSLs

The domain model of your application is built out of intention-revealing abstractions. The DSL layer that you offer on top of the model as a facade becomes usable and extensible only if the abstractions are at the correct level. In this section, we’ll consider the whole DSL as an abstraction and discuss how you can compose DSLs together. This technique comes in handy when you need to weave multiple Scala DSLs together and compose a bigger whole. In our domain of designing trading systems, integrating market-rule DSLs with the core business rules of trade processing is one use case that we’ll be discussing.

After you have a DSL that’s designed as an abstraction, you can extend it through subtyping in Scala. Subtyping can lead to an entire DSL hierarchy, with each of the specialized abstractions providing different implementations to the same core language. Sound polymorphic? Sure they are, and we’ll use polymorphism to compose DSLs in section 6.7.1. We’ll also look at composing unrelated DSLs in section 6.7.2; after all, DSLs tend to evolve independently of each other and of your application lifecycle. Your application architecture must be capable of hosting a seamless composition of multiple DSL structures.

6.7.1. Composing using extensions

When a trade has been entered into the system, it passes through a normal trading lifecycle that begins with enrichment. This process adds some of the derived information to the trade record that didn’t originally come from the upstream system. This information includes the cash value of the trade, applicable taxes and fees, and other components that vary with the type of instrument being traded.

Growing up the DSL

Consider the following DSL snippet. It doesn’t look like a DSL right now, but you are going to add more meat to its bones as we move along. You are also going to define some of the trade lifecycle methods using the components that you’ve implemented so far.

There’s nothing semantically rich about the language at the moment. It just defines a method enrich, which is supposed to enrich a trade after it’s been entered into the system .

Let’s define specific implementations of TradeDsl for FixedIncomeTrade and EquityTrade in listings 6.13 and 6.14. The DSL for FixedIncomeTrade uses the FixedIncomeTradingService abstraction that we designed earlier.

Listing 6.13. Trade DSL for FixedIncomeTrade

The DSL for EquityTrade uses the EquityTradingService abstraction that we defined earlier.

Listing 6.14. Trade DSL for EquityTrade

package dsl

import api._
trait EquityTradeDsl extends TradeDsl {
  type T = EquityTrade

  import EquityTradingService._

  override def enrich: PartialFunction[T, T] = {
    case t =>
      t.cashValue = cashValue(t)
      t.taxes = taxes(t)
      t
  }
}

object EquityTradeDsl extends EquityTradeDsl

In listings 6.13 and 6.14, we have FixedIncomeTradeDsl and EquityTradeDsl as individual concrete languages that implement the same core language of TradeDsl. To implement the enrichment semantics, they use the TradingService implementations that we designed in section 6.6. The class diagram in figure 6.6 shows how the two language abstractions are related to each other.

Figure 6.6. TradeDSL has an abstract type member T <: Trade, but EquityTradeDSL has the concrete type T = EquityTrade and FixedIncomeTradeDSL has the concrete type T = FixedIncomeTrade. TradeDSL has two specializations in EquityTradeDSL and FixedIncomeTradeDSL.

Because FixedIncomeTradeDSL and EquityTradeDSL are extensions of the same base abstraction, they can be used polymorphically through the usual idioms of inheritance. But consider yet another type of TradeDSL that’s not specialized on the type of the trade. It models another business rule that needs to compose with the semantics of both EquityTradeDSL and FixedIncomeTradeDSL. Let’s illustrate this composition technique in Scala using an example.

Composing DSLs with pluggable semantics

Business rules change with changes in market conditions, regulations, and lots of other factors. Let’s assume that the stock broker organization announces a new market rule for promoting high value trades as follows:

• “Any trade on the New York Stock Exchange of principal value > 1000 USD must have a discount of 10% of the principal on the net cash value.”

Now this rule needs to be implemented when we enrich the trade, irrespective of whether its type is EquityTrade or FixedIncomeTrade. You don’t want to include it as part of the core cash value calculation; it’s a promotional market rule that shouldn’t impact the core logic of the system. Rather, you should implement such domain rules like the layers of an onion so you can include and exclude them flexibly without intruding into your core abstractions (think decorators). Let’s extend our TradeDsl with some new semantics that reflect these market-specific rules in the following listing.

Listing 6.15. A new semantics for TradeDsl—another business rule

This is the exciting part, where we compose the DSLs. Note the abstract val semantics that embeds the DSL that we want to be composed with this new domain rule . Internal DSLs are also known as embedded DSLs. But in most cases, you’ll find that the semantics are hardwired within the implementation of the DSL. In this particular case, we want to make the actual semantics of the composed DSL pluggable. By making them pluggable, you have the loose coupling between the composed DSLs. At the same time, runtime pluggability lets you defer your commitment to the concrete implementation. In the following listing, you define concrete objects for EquityTradeDsl and FixedIncomeTradeDsl, composed with the new MarketRuleDsl.

Listing 6.16. DSL composition

package dsl

object EquityTradeMarketRuleDsl extends MarketRuleDsl {
  val semantics = EquityTradeDsl
}

object FixedIncomeTradeMarketRuleDsl extends MarketRuleDsl {
  val semantics = FixedIncomeTradeDsl
}

Later you’ll look at the entire set of composed DSLs in action. But first let’s add more functionality to TradeDsl using some of your knowledge about functional combinators. Combinators will give us compositional semantics both at the functional level and at the object level.

More composition with functional combinators

Remember we talked about the trade lifecycle earlier in this section? Before the trade is enriched, it is validated. After the enrichment is done, it is journalized to the books of the accounting system. A few more steps occur in the real-world application, but for the purpose of demonstration, let’s keep it short for now. How can you model this business rule in a Scala DSL?

You will use PartialFunction combinators to model this sequencing, and pattern matching to make the rule explicit. The following listing enriches our original implementation of TradeDsl and adds a control structure that models this business rule.

Listing 6.17. Modeling the trade lifecycle in a DSL

You must have been wondering why you defined enrich as a PartialFunction. Partial functions in Scala use the amazing power of composition to build higher order structures.

You have defined a control structure withTrade that takes an input trade and lets you perform the complete sequence of lifecycle operations on it. This control structure also has an option to add custom operations to the trade lifecycle in the form of an additional argument (op: T => Unit) . This argument is a function that operates on the trade but returns nothing. One typical use of such functions is to add side-effecting operations to the trade lifecycle. Logging, sending emails, and doing audit trails are examples of functions that have side effects but that don’t alter the return value of the final operation.

Now let’s look into the pattern-matching block within withTrade. The entire domain rule is expressed within the four lines of code that it contains. The andThen combinator also nicely expresses the sequence that the trade lifecycle needs to follow.

Using the fully composed DSL

The following listing shows the whole composition in action. This DSL creates a trade using our trade-creation DSL, does all sorts of enrichment, validation, and other lifecycle operations, and finally composes with the market rules DSL to generate the final cash value of the trade.

Listing 6.18. The trade lifecycle DSL

import FixedIncomeTradeMarketRuleDsl._

withTrade(
  200 discount_bonds IBM
    for_client NOMURA
      on NYSE
        at 72.ccy(USD)) {trade =>
  Mailer(user) mail trade
  Logger log trade
} cashValue

You used the Decorator design pattern as your composition technique in this section (see [3] in section 6.10). We consider the semantics to be the decorated DSL; the wrapper provides the necessary decoration. You can use the Decorator pattern to implement dynamic inclusion and exclusion of responsibilities from an object. No wonder it turned out to be a useful tool here, when we needed to compose families of DSLs together.

What happens if the languages that you want to compose aren’t related? Frequently, you’ll use utility DSLs for modeling date and time, currencies, and geometric shapes that find applicability within the context of other larger DSLs. Let’s see how you can manage their evolution seamlessly.

6.7.2. Composing different DSLs using hierarchical composition

It is quite common to use a smaller DSL embedded within a larger one. Trading solutions can use DSLs for expressing currency manipulations and conversions, date and time management, and customer balance management in portfolio reporting, to name a few situations in which you’ll find them.

Now suppose you’re implementing a DSL for client portfolio reporting. You need to report balances for securities and cash holdings that the client account holds as of a particular date. Note the two italicized words: client-portfolio and balance represent two important domain concepts and are candidates for DSL-based implementations. They’re independent abstractions, but they often occur in close association with each other.

Avoiding implementation coupling

Let’s find out in table 6.7 how you can make this association clean enough so that the individual DSLs can evolve as independent implementations.

Table 6.7. Composing DSLs hierarchically

Balance is the interface that abstracts the underlying implementation. Scala lets you define type synonyms. You can define type Balance = BigDecimal and happily use Balance as the name to describe the net value of your client holdings in his portfolio. But what happens when you build larger families of Portfolio DSLs as specializations of your base DSL, just like the ones we made with TradeDsl in section 6.7.1? Embedding the implementation of Balance within the Portfolio DSL will couple the entire hierarchy to a concrete implementation. Even if you need to, you’ll never be able to change the implementation in the future. The solution is to avoid directly embedding one DSL in the other and instead compose them hierarchically. In the end, you’ll have two DSLs that fit together nicely and that are coupled loosely enough so that you can plug in your implementation of choice whenever you feel like it.

Consider the following DSL for modeling your client’s portfolio in the following listing.

Listing 6.19. A DSL with implementation coupling

Ugh! By the time you get down to the first specialization of the Portfolio DSL, you see that the Balance abstraction has already been broken through . Try to compose them hierarchically and keep the implementation of the Balance DSL outside the Portfolio DSL. Composing hierarchically means that one needs to be within the other. The difference between composing hierarchically and the code in listing 6.19 is that you are going to embed the DSL interface and NOT the implementation. I can almost hear the murmur of abstract vals coming across from my readers. You guessed it! The Portfolio DSL needs to have an instance of the Balance DSL, which we’ll call Balances.

How do you model balance?

To get a true understanding of a DSL, we can’t deal with examples that are too trivial. After all, you can appreciate the expressiveness of a DSL only when you realize how it abstracts the underlying complexities in a readable syntax. Initially we talked about modeling a balance with a BigDecimal. But if you’re a domain person familiar with the securities trading operations, you know that for a client account, the balance indicates the client’s cash position in a particular currency as of a specific date. I’m not going into the details of how you compute balance from a client portfolio. Modeling a balance with only a BigDecimal is an oversimplification. The Balances DSL contract is shown in the following listing, followed by a sample implementation BalancesImpl.

Listing 6.20. DSL for modeling account balance

A client balance can be reported in a specific currency, depending on the client’s preference. But for auditory regulations, it’s often required to be converted to a base currency. A base currency is one in which the investor maintains its book of accounts. In the forex market, the US dollar is usually considered to be the base currency. In the DSL shown in listing 6.20, the method inBaseCurrency reports the balance in the base currency. In the sample implementation of Balances, we commit to an implementation of the abstract type Balance as a tuple of three elements: the amount, the currency, and the date (a balance is always calculated as of a specific date).

Composing the Balance DSL with the Portfolio DSL

In order to compose with the Portfolio DSL, you need an abstract val of Balances as a data member within it, as shown in the following listing.

Listing 6.21. The Portfolio DSL contract

Note the object import syntax of Scala that makes all members of the object bal available within the class body. Now let’s look at a specialized implementation of Portfolio that computes the balance of a client account in the next listing.

Listing 6.22. A DSL implementation of Portfolio

We’ve committed to a specific implementation of Balances in our ClientPortfolio DSL. Now we need to ensure that when we compose ClientPortfolio with other DSLs that use Balances, those other DSLs will also use the same implementation.

Let’s look at another implementation of Portfolio that acts as the decorator of other Portfolio implementations. In the following listing we look at Auditing, an implementation of Portfolio that adds auditing features to other Portfolio implementations.

Listing 6.23. Another implementation of the Portfolio DSL

Auditing not only composes with another Portfolio DSL , but it also ensures that the DSL it embeds within itself (semantics) uses the same implementation of the embedded Balances DSL . (Balances is embedded within Portfolio, which is the superclass of Auditing.) We enforce this constraint by declaring bal in Auditing to be semantics.bal, which defines it as a Scala singleton type. Now we can specify the concrete implementation values of semantics and bal to create an abstraction for ClientPortfolio that supports Auditing. Look at the following snippet:

object ClientPortfolioAuditing extends Auditing {
  val semantics = ClientPortfolio
  val bal: semantics.bal.type = semantics.bal
}

When you use hierarchical composition to compose multiple DSLs, you get the advantages listed in table 6.8.

Table 6.8. Advantages of using hierarchical composition for DSLs

The subject of DSL composition is well explained in the paper Polymorphic embedding of DSLs (see [7] in section 6.10). Read the paper if you want to get into the details of other ways to compose DSLs in Scala.

In earlier sections of this chapter, you’ve seen lots of techniques you can use to compose abstractions using the object-functional power of Scala. The discussion of composition is still incomplete, because we haven’t talked about monadic abstractions. Monadic abstractions are used extensively to build composable computations. Monads have their origin in category theory (don’t worry, I’m not going to discuss the mathematics behind monads; you can look into that on your own if you’re interested). In the next section, you’re going to see how you can put some syntactic sugar on top of monadic structures in Scala. This additional magic will help you design sequencing of DSL operations.

6.8. Monadic structures in DSL

I’ve probably put enough emphasis on how abstractions that compose well make readable DSLs. When we talk about abstracting a computation, functional programming gives you more composing power than the OO programming model. You get this added boost because functional programming treats computations as applications of pure mathematical functions, without the side effects of mutable state. The moment you decouple functions from mutability, all you’re left with are abstractions that you can verify independently, without any dependence on external context. Functional programming offers mathematical models that let you compose computations as functional compositions. I’m not going into the depths of category theory or any similar formalism that makes this promise. The only thing you need to remember is the fact that compositionality of functions gives you a way to form complex abstractions out of simpler building blocks.

What’s a monad?

You can think of monads as function composition, and binding on steroids. When you build abstractions that obey the monad laws, you can use them to construct higher-order abstractions by using beautiful composition semantics

A monad is an abstraction in which you structure your computation into values and sequences of computations that use them. You can use them to compose dependent computations to form larger computations. Monads have their theoretical basis in category theory, which might be something you don’t want to contemplate right now. If, on the other hand, you’re not scared of category theory, go read [11] in section 6.10. For the rest of you, relax. We’re not going to get into the guts of that theory here.

We’re going to examine monads only in terms of what you need from them to design a Scala DSL. You’ll see a lot more monadic structures when we discuss parser combinators in Scala in chapter 8. In this section, we’re going to talk about some of the monadic operations in Scala that can make your DSLs compose more beautifully than when you use an OO counterpart.

The accompanying sidebar gives a brief introduction to monads. For more details about monads, refer to [9] in section 6.10.

How monads reduce accidental complexity

Consider the Java example in listing 6.24 of a typical operation in your web-based trading application. You have a key that you use to get the value from HttpRequest or HttpSession. The value that you get is the reference number of a trade. You need to do a query for the trade from the database to get the corresponding Trade model.

Listing 6.24. Handling alternate routes in computation using Java

This code shows some nonessential complexity , plaguing the surface syntax of your abstraction. All the null checks have to be done explicitly and at every step in this computation of a domain model from a context parameter. Now consider the following equivalent Scala code that does the same thing as the previous code using its monadic for comprehension syntax.

Listing 6.25. Scala monadic for comprehensions

Let’s look more into how the monadic structures in this listing chain together to create the trade object within the main method.

param returns a monad Option[String] . Option[] is a monad in Scala that you can use to abstract a computation that might not produce any result. queryTrade returns another monad, Option[Trade] , which has a type that’s different from Option[String]. We need to chain these computations such that if param returns a null value, queryTrade must not be invoked. We did this check explicitly in listing 6.24. Using monadic structures, the underlying implementation of the monad Option[] takes care of this plumbing so that your code remains clean and free of accidental complexity .

How does a monad take care of this chaining? It’s through the bind operation that we discussed in the monad laws in the sidebar earlier in this section. In Scala, bind is implemented as flatMap; the for comprehension is just syntactic sugar on top of flatMap, as you can see in the following snippet.

Here’s the unsweetened version of the for comprehensions that makes this bind explicit.

param("refNo") flatMap {r =>
  queryTrade(r) map {t =>
    t}} getOrElse error("not found")

flatMap (equivalent to the >>= operation in Haskell) is a combinator that serves as the glue for this snippet. It’s the overarching bind operator that injects the output of param into the input of queryTrade, handling all the necessary null checks within it. For comprehensions offer a higher level abstraction on top of the flatMap combinator to make your DSL more readable and expressive.

A detailed discussion of monads, flatMaps, and for comprehensions is beyond the scope of this book. What you need to know is that monadic structures and operations give you an easy way to implement DSLs. We’ve just looked at one of the most common uses of monads as a means of sequencing dependent computations without introducing nonessential complexities. Apart from this, monads are also used extensively as a mechanism to explain side effects in pure functional languages, handle state changes, exceptions, continuations, and many other computations. Needless to say, you can use all of these ways to make your DSLs appear more expressive. For more details about monads in Scala, see [10] in section 6.10.

To tie a nice big bow on our discussion about how monads help make expressive language abstractions, let’s build one variant of the trade lifecycle DSL that we implemented in listing 6.20. Instead of using partial functions to sequence operations, we’ll use monadic comprehensions of Scala. Doing this exercise will give you an idea about how to think in terms of monads when building your own DSL. We’ll keep the example simple just to demonstrate how you can build your own computations that can be chained using for comprehensions.

Designing a monadic Trade DSL

Without much ado, let’s define a variant of the TradeDSL that we discussed in listing 6.17. Each of the lifecycle methods now returns a monad (Option[]) instead of PartialFunction.

Listing 6.26. Monadic TradeDsl

package monad

import api._
class TradeDslM {
  def validate(trade: Trade): Option[Trade] = //..
  def enrich(trade: Trade): Option[Trade] = //..
  def journalize(trade: Trade): Option[Trade] = //..
}

object TradeDslM extends TradeDslM

We can use that DSL within a for comprehension to invoke the sequence of lifecycle methods on a collection of trades:

import TradeDslM._

val trd =
  for {
    trade <- trades
    trValidated <- validate(trade)
    trEnriched <- enrich(trValidated)
    trFinal <- journalize(trEnriched)
  }
  yield trFinal

This snippet has the same functionality as listing 6.20 but it uses monadic binds to chain operations. In the case of our earlier implementation that was based on partial functions, we could only chain operations that matched exactly in types. In this for comprehension, there are sequenced operations that don’t have the types that match exactly. trades is a List of Iterable that generates a Trade for every iteration of the comprehension execution. We don’t have to check for the end-of-sequence explicitly because List is implemented as a monad; just like Option[], the flatMap combinator within List takes care of such boundary conditions. validate returns an Option[Trade], which can be Some(trade) or None. When we pipeline the output of validate into enrich, we don’t do any explicit null checks or any explicit conversion from Option[Trade] -> Trade. So long as you pipeline using monadic structures like List[] or Option[], all binds are done automatically through the flatMap combinator. In this sense, chaining operations using monadic binds is more powerful than what we achieved using partial functions in listing 6.17. When they’re designed correctly, monadic operations can lead to expressive DSLs, especially if you use the syntactic sugar that for expressions provide in Scala (or the do notation in Haskell).

In case you’re curious, the previous snippet boils down to the following flatMap expression:

trades flatMap {trade =>
  validate(trade) flatMap {trValidated =>
    enrich(trValidated) flatMap {trEnriched =>
      journalize(trEnriched) map {trFinal =>
        trFinal
      }
    }
  }
}

Obviously, this looks more programmatic and less readable to the domain user than the earlier version that uses for expressions.

As you saw in this section, monads in Scala are yet another way to compose abstractions. They’re subtly different from partial functions and offer a truly mathematical way to chain dependent abstractions. Scala offers quite a few built-in monadic structures. Be sure to use them appropriately when you implement DSLs in Scala.

6.9. Summary

The Scala community is abuzz with DSLs and not without reason. Scala is one of the most potent forces among the programming languages today. It offers first-class support for designing expressive DSLs.

In this chapter, you’ve taken a thorough tour of all the Scala features that help you design internal DSLs. We started with a summary of the feature list of Scala, then carefully dug deep into each of them by examining snippets of DSLs designed from the domain of securities trading. A DSL is architected as a facade on top of an underlying implementation model. In this chapter, we switched back and forth between the domain model and the language abstractions that are on top of it.

DSLs need to be composable at the contract level, without exposing any of their implementation details. You have to design them this way because every DSL evolves on its own and you might need to change an implementation without impacting other DSLs or your core application. You saw how internal DSLs in Scala compose together statically using the power of the type system. Finally, we took a tour of how monadic operations can help construct composable DSL structures. Scala doesn’t make monads as explicit as Haskell does in its programming model. But you can still use Scala’s monadic comprehension to sequence domain operations without introducing a lot of nonessential complexities.

Internal DSLs are hosted within a specific language and are sometimes limited by the capabilities of the host language. You can get around these limits by designing your own external DSL. In the next chapter, we’re going to look at some of the building blocks of external DSLs. We’ll start with some of the basic theories of compilers and parsers, and then move on to higher-order structures like parser combinators that are widely used today. Stay tuned!

6.10. References

1. Odersky, Martin, and Matthias Zenger. 2005. Scalable component abstractions. Proceedings of the 20th annual ACM SIGPLAN conference on object-oriented programming systems, languages, and applications, pp 41-57.
2. Wampler, Dean, and Alex Payne. 2009. Programming Scala: Scalability = Functional Programming + Objects. O’Reilly Media.
3. Gamma, E., R. Helm, R. Johnson, and J. Vlissides. 1995. Design Patterns: Elements of reusable object-oriented software. Addison-Wesley Professional.
4. Ford, Neal, Advanced DSLs in Ruby, http://github.com/nealford/presentations/tree/master.
5. Emir, Burak, Martin Odersky, and John Williams. Matching Objects With Patterns. LAMP-REPORT-2006-006. http://lamp.epfl.ch/~emir/written/Matching-ObjectsWithPatterns-TR.pdf.
6. Evans, Eric. 2003. Domain-Driven Design: Tackling complexity in the heart of software. Addison-Wesley Professional.
7. Hofer, Christian, Klaus Ostermann, Tillmann Rendel, and Adriaan Moors. Polymorphic Embedding of DSLs. Proceedings of the 7th international conference on generative programming and component engineering, 2008, pp 137-148.
8. ScalaTest. http://www.scalatest.org.
9. Wadler, Philip. 1992. The essence of functional programming. Proceedings of the 19th ACM SIGPLAN-SIGACT symposium on principles of programming languages. pp 1-14.
10. Emir, Burak. Monads in Scala. http://lamp.epfl.ch/~emir/bqbase/2005/01/20/monad.html.
11. Pierce, Benjamin C. 1991. Basic Category Theory for Computer Scientists. The MIT Press.
12. Ghosh, Debasish. Implementation Inheritance with Mixins—Some Thoughts. Ruminations of a Programmer. http://debasishg.blogspot.com/2008/02/implementation-inheritance-with-mixins.html.
13. Venners, Bill. Abstract Type Members versus Generic Type Parameters in Scala. http://www.artima.com/weblogs/viewpost.jsp?thread=270195.
14. Ghosh, Debasish. Scala Self-Type Annotations for Constrained Orthogonality. Ruminations of a Programmer. http://debasishg.blogspot.com/2010/02/scala-self-type-annotations-for.html.