C H A P T E R  9

Stitch — A DSL for Hosting Languages

Back in Chapter 6, we looked at the DLR Hosting API. In that chapter, we saw that the Hosting API provides a uniform way for a host language like C# to embed languages like IronPython and IronRuby. Although the Hosting API provides a good layer of abstraction between host and hosted languages, there's still room for pushing the level of abstraction even higher—and that's the topic of this chapter. We'll go through the design and implementation of a domain specific language (DSL) called Stitch that I developed for language hosting. Using Stitch, we can host not just DLR-based languages but also languages like Windows PowerShell—uniformly and declaratively. The set of languages Stitch is capable of hosting is extensible. Moreover, the runtime of the Stitch language provides execution modes for running code sequentially or in parallel.

The Stitch language is so named because it is built to make it easy and painless to stitch together other languages. It's a DSL, not a general purpose language, because it is designed to solve only the issues encountered in language hosting and nothing else.

The Need for Stitch

In Chapter 6, we saw examples like the one in Listing 9-1 that use the DLR Hosting API to host IronPython code in C#. The nice thing about these examples is that if we change the language we're hosting from IronPython to IronRuby, we don't need to code against a different hosting API. Our code will still use the ScriptRuntime class to get an instance of ScriptEngine. It will still use an instance of the ScriptScope to pass information to the IronRuby code.

Although the DLR Hosting API provides a language-agnostic API for hosting multiple languages, there are some issues in the Listing 9-1 code, as the following list describes.

The DLR Hosting API is imperative: It's very imperative because the code we write instructs how to run IronRuby or IronPython code by creating instances of ScriptRuntime and ScriptEngine, calling methods on those instances, fishing out objects from script scopes, and so on and so forth. Instead of doing all of this, it would be nice if we could be declarative and avoid having to tell the Hosting API how to execute IronPython or IronRuby code every step of the way.

The DLR Hosting API is platform-dependent: The code examples we saw in Chapter 6 can run only on .NET using the DLR runtime. The Hosting API can't be used to run the same Python, Ruby or other dynamic language scripts on a different platform, such as the JVM. It would be nice if we could write the code once and run it on different platforms.

The DLR Hosting API serves only DLR-based languages: It can't host languages such as PowerShell, Ant, Maven, etc. Developers still need to know how to use PowerShell from IronPython or how to use IronPython from PowerShell, or how to use PowerShell from Ruby or vice versa. We don't want to learn the different ways of using one language within another. We want a generic approach that works for all scenarios.

The DLR Hosting API provides no high level support for parallel execution of scripts: We can use Parallel Extensions. But it would be nice if there were a higher level of abstraction for hosting languages.

The design goal of the Stitch DSL is to address these four issues. Let's see the solution Stitch provides.

Syntax of the Stitch Language

Let's look at the Stitch language syntax now and see how it addresses the four issues we just mentioned. When I started out designing Stitch's syntax, I wrote down something like this:

The syntax prototype uses <Python> to introduce a block of Python code and <Ruby> a block of Ruby code. With the syntax prototype, you can stitch together multiple pieces of Python code and Ruby code in any order you like. The syntax is declarative because it does not specify how to run the Python or Ruby code. It only indicates which block is Python code and which block is Ruby code. With this syntax prototype, how to run the Python and Ruby code becomes the job of the Stitch language runtime, not the job of developers who write Stitch code.

The syntax prototype is not only declarative, it's also platform-independent. It's not tied to anything .NET or DLR-specific. If we implement a Stitch runtime that runs on the Java virtual machine, then any Stitch code we write can also run on the Java virtual machine.

To run code written in languages that are not DLR-based, such as PowerShell, I extended the syntax prototype to something like the following:

So far, everything seems pretty simple. The devil, of course, is in the details. The different language code snippets we stitch together are not islands. To do interesting things, they often need to send information to each other like this:

This syntax prototype conveys the idea that the variables x and z defined in the Python code are accessible to the Ruby code that follows. The syntax looks good except for a couple of issues. First, the Ruby code has a dependency on the Python code because it uses the variable x from the Python code. The dependency is not very explicit. To find the dependency, we have to scan through all the lines of code and analyze them. Another issue with the syntax prototype is that the variable z in the Python is accessible to but not used in the Ruby code. This is not desirable because the Ruby code might unknowingly define a variable by the same name z and accidentally change the value of the Python variable z. To solve those two issues, I changed the syntax prototype to something like this:

This syntax prototype basically says that the Python code takes no input parameters and returns x and z as results. The Ruby code takes x as an input parameter and returns no results. With this syntax, the variable z defined in the Python code will not be accessible to the Ruby code because the Ruby code does not declare z to be its input parameter. The syntax also makes the dependency between the Ruby code and Python code explicit. To find the dependency, we only need to see which variables the Python code returns and which input parameters the Ruby code requires. Once the dependencies between different language code snippets are explicit, the Stitch language runtime can easily tell which code snippets can run in parallel. For example, if the Ruby code did not require the variable x as an input parameter, the Stitch runtime would detect no dependency between the Ruby code and the Python code and it would therefore run the two code snippets in parallel.

At this point, the design of Stitch's syntax solves all of the four issues I set out to address. One last thing I want to do is to make the syntax look as natural as possible to developers who write Stitch code. To that end, I changed the syntax to something like this:

Instead of <Python()>, the new syntax uses <foo () Python>. The word foo is a unique identifier we attribute to a language code snippet. The language name Python is moved from before the input parameter list to after the list. This change makes the syntax look more like a C# method definition. In Stitch terms, a language code block is called a function. In the previous code example, there are two functions—foo and bar. The foo function is a Python code block that takes no input parameters and returns the variable x as the result. The bar function is a Python code block that takes no input parameters and returns the variable y as the result. Another change is the new include syntax keyword for including code from a file. In the example above, the include keyword will include the code in the file baz.py. The Stitch code example declares that the code in baz.py takes no input parameters and returns the variable y as result. When using the include keyword, we don't need to specify which language the included code is written in because the file extension of baz.py already indicates that the language is Python.

That's all for the syntax of the Stitch language. Next I'll explain how to set up the software components required for running the code examples in this chapter. After that, we'll look at four examples of Stitch code that demonstrate the core features of the Stitch language.

Requirements for the Example

To follow along with the code examples in this chapter, you'll need to install some software. You'll also need to understand how I've organized the code for this chapter in the example download for this book.

Software Requirements

To follow along this chapter's code examples, you'll need to download and install Windows PowerShell and the C# runtime of ANTLR. Here's what to do:

1. Download and install PowerShell. The version I use is 1.0. Instructions for downloading are available from:
   www.microsoft.com/windowsserver2003/technologies/management/powershell/download.mspx.
2. Download the C# runtime of ANTLR from
   www.antlr.org/download/CSharp. The file I downloaded is DOT-NET-runtime-3.1.3.zip. Unzip the file into C:ProDLRlibAntlr.

After completing these steps, you should have the following two assemblies on your hard disk:

• C:ProDLRlibAntlrin et-2.0Antlr3.Runtime.dll
• C:Program FilesReference
  AssembliesMicrosoftWindowsPowerShellv1.0System.Management.Automation.dll

These will be the only assemblies you need for the examples in this chapter.

Organization of the Code

The code for this chapter is organized into four projects:

The Eclipse Stitch project—this project has only a file called Stitch.g. The file defines the grammar of the Stitch language. I wrote the grammar definitions in the ANTLR grammar language and used ANTLR to generate the lexer and parser (in C# code) for Stitch. ANTLR is a popular software component for defining language grammars and for generating lexers and parsers. The lexer and parser generated by the grammar file are put into the C# Stitch project.

The C# Stitch project—this project has the implementation of the Stitch language's runtime as well as Stitch's language plug-in framework.

The PowerShellStitchPlugin project—this project implements the Stitch language plug-in for the PowerShell language.

The StitchDemoApplication project—this project is the client program that hooks up the PowerShell plug-in with the Stitch language runtime. The project contains the test Stitch scripts you saw earlier in this chapter.

In the sections that follow, we will begin to explore the Stitch language implementation by first looking at some examples of how Stitch is used. Then we'll look at the language grammar. After that, we'll examine the language plug-in framework and the core Stitch language runtime.

Stitch in Use

Now that you've set up the required software components for this chapter, let's run some examples to get a feel for Stitch's language features. We are going to look at four examples. In this chapter's Visual Studio solution, you'll find the code for all four examples in the Scripts folder of the StitchDemoApplication project. The first example, testScript1.st, demonstrates the declarative aspect of the Stitch language. The second example, testScript2.st, shows that Stitch is capable of hosting non-DLR based languages. The third example, testScript3.st, shows that Stitch scripts can be hosted within other Stitch scripts. The fourth example, testScript4.st, hosts several Python scripts and executes them both sequentially and in parallel. Let's begin.

Being Declarative

Stitch is a declarative language, not an imperative one. When writing Stitch code, you express what you want to get done, not how you want it to get done. Listing 9-2 shows the code for our first Stitch example, which demonstrates the declarative nature of the Stitch language. The code consists of three Stitch functions—arithmetic, addition, and calculation. The arithmetic function is a Python code block that defines the myadd Python function. The addition function includes the add3.py script file. If you open add3.py, you'll see that it has only one line of Python code that defines a function called add3: def add3(x): return x + 3. The calculation function takes the myadd and add3 functions as input and uses them to do some simple calculations. As you can see, the code in Listing 9-2 is declarative, exactly what I want to demonstrate.

To run the code in Listing 9-2, you can use the C# code in Listing 9-3, which you'll find in the Program.cs file of the StitchDemoApplication project. The C# code creates an instance of the StitchScriptEngine class and tells the Stitch script engine that we want to run the Stitch code in the sequential mode. Later in this chapter, we'll see the different modes for executing Stitch code and how they are implemented. In Listing 9-3, the code passes a PowerShellPlugin instance to the constructor of StitchScriptEngine so that the Stitch script knows how to stitch PowerShell code with other language code blocks. Later in the chapter we'll see the plug-in framework of the Stitch language and how you can use it to add support for new languages into Stitch. When you run the code in Listing 9-3, you'll see two numbers, 7 and 5, printed on the screen.

Hosting DLR- and Non-DLR-Based Languages

Unlike the DLR Hosting API, the Stitch language allows you to host both DLR and non-DLR-based languages. The code in Listing 9-4 shows a Stitch example that hosts a PowerShell code block and a Python code block. The PowerShell code block is the function called getServiceA. It contains one line of PowerShell code that returns all Windows services whose name begins with “A”. The Python code block is the printServiceA function, which takes the Windows services returned by the getServiceA function and prints the names of those Windows services to the screen.

As in the previous example, we will use some C# code to run the Stitch script in Listing 9-4. Since the C# code is almost the same as the code in Listing 9-3, I won't show it here, but you can find it in the RunTestScript2 method inside the Program.cs file of the StitchDemoApplication project. When you run the code, you'll see the names of the Windows services that are installed on your machine whose name begins with “A”.

Hosting Stitch Itself

When we stitch pieces of cloth into larger pieces, it seems perfectly natural that we can further stitch those larger pieces into even larger pieces. The Stitch language allows us to stitch code just as we'd stitch pieces of cloth. The code in Listing 9-5 shows a Stitch example that stitches the two Stitch scripts, testScript1.st and testScript2.st, from the previous two sections.

To run the code in Listing 9-5, you can use the C# code in the RunTestScript3 method inside the Program.cs file of the StitchDemoApplication project. When you run the code, you'll see the combined results of the previous two examples printed on the screen.

Executing in Parallel

The Stitch language runtime provides execution modes for running Stitch code in either a sequential manner or in parallel. Later in this chapter, you will see that the implementation of the Stitch language runtime uses .NET Parallel Extension to run Stitch code in parallel. For now, let's just see an example that shows the parallel execution feature of Stitch in action. The code in Listing 9-6 has three Stitch functions—task1, task2, and task3. Each of the functions consists of a block of Python code that prints a message to the screen and calls Thread.SpinWait to simulate some busy work. The three Stitch functions don't have any dependency among them and therefore are perfectly ideal for parallel execution.

The C# code in Listing 9-7 runs the Stitch script in Listing 9-6 twice, first in a sequential manner and then in a parallel manner. To run a Stitch script sequentially, you specify the execution mode to be ExecutionMode.Sequential when you create the StitchScriptEngine instance. To run a Stitch script in parallel, you specify the execution mode to be either ExecutionMode.ParallelNoWait or ExecutionModel.ParallelWaitAll. If you specify ExecutionMode.ParallelNoWait, the thread executing the RunTestScript4 method will continue to run without waiting for the three Stitch functions in Listing 9-6 to finish. On the other hand, if you specify ExecutionMode.ParallelWaitAll, the thread executing the RunTestScript4 method will be suspended until all three of the Stitch functions complete. We will see the implementation details of the three execution modes later in the chapter. If you run the code in Listing 9-7 on a multi-core CPU machine, you'll see that in the first run of testScript4.st, all of the three Stitch functions task1, task2, and task3 run on the same thread (the same thread id is printed on the screen). The second run of the Stitch script is much faster as the three Stitch functions run on different threads (different thread ids are printed on the screen).

Stitch Language Grammar

We saw the Stitch language syntax in the “Syntax of the Stitch Language” section. Now we are going to formally define the grammar of the Stitch syntax and use the grammar to generate the lexer and parser for the Stitch language. Because I wrote the grammar in the ANTLR grammar language and used Eclipse as the IDE for writing the grammar, if you want to follow along this part of the Stitch language implementation, you'll need to set up some software components. ANTLR is a popular lexer/parser generator and more. There are many lexer/parser generators out there and people often prefer certain ones over others. Because of that and also because I want to keep this chapter's focus on the DLR Hosting API, I won't dive too deeply into how ANTLR works. A fair coverage of a lexer/parser generator can easily grow into several chapters and that's really beyond the scope of what I'm aiming to cover here.

If you want, you can completely skip all the ANTLR-related parts and simply take the generated lexer/parser C# code as given. For those who would like to follow this chapter's code from the very beginning, I'll describe the software components you need to install.

Setting Up Eclipse and ANTLR

I used the following software components to develop the ANTLR-based grammar for the Stitch language.

• Java Development Kit (JDK) 6.
• Eclipse 3.6.0.
• ANTLR 3.2
• ANTLR IDE 2.1.0

You should already have Java and Eclipse installed, or know how to get them. Here are the steps to follow to install the two ANTLR components:

1. Download the complete ANTLR 3.2 jar file (antlr-3.2.jar) from www.antlr.org and place it in the C:antlr-3.2lib folder. If you put it in a different folder, you'll need to substitute the file path with your own when I refer to C:antlr-3.2.
2. Install ANTLR IDE 2.1.0 and its prerequisites. You can download it from the ANTLR IDE project web site at http://antlrv3ide.sourceforge.net/. According to the download page of the web site, ANTLR IDE 2.1.0 requires GEF 3.6.0 or above, Zest 1.2.0 or above, and Dynamic Language Toolkit (DLTK) Core 2.0.0.

After you've downloaded and installed all of these components, you need to set them up properly by configuring the Eclipse IDE to use the ANTLR 3.2 jar file. To do so, launch Eclipse and select Window->Preferences in the menu. In the Preferences dialog that pops up, expand the ANTLR node in the tree on the left and select the Builder node under the ANTLR node. In the right panel, click the Add… button and you'll see a dialog like the one in Figure 9-1. All of the fields in the second dialog will be blank at this point, because you haven't told Eclipse about the ANTLR 3.2 jar file you want to use. In the second dialog, click the Directory… button and another dialog will pop up. Select C:antlr-3.2 as the folder and click OK. At this point, you should see something that what's shown in Figure 9-1.

Figure 9-1. The dialog for telling Eclipse which ANTLR.jar file to use

Once you've told Eclipse which ANTLR jar file to use, you can write ANTLR-based language grammars. For the Stitch language, I created a Java project called Stitch in Eclipse, then I converted the project into an ANTLR project. To do that, right-click on the Stitch project in Eclipse and select Configure->Convert to ANTLR Project… in the context menu. With the Stitch project in place, the next step is to create a grammar file.

Defining the Grammar

To create a grammar file, right-click on the src folder in the Stitch project and select New->Other… in the context menu. A dialog will pop up. This is the wizard for creating a new file in Eclipse. The first step is to select the type of file you want to create, in this case an ANTLR combined grammar file as shown in Figure 9-2.

Figure 9-2. The Eclipse new-file creation wizard

Click Next and select CSharp2 as the language. This causes ANTLR to generate the lexer/parser code in C# later. Name the grammar file Stitch.g. Once Stitch.g is created, open it in the Eclipse IDE and paste the contents of C:ProDLRsrcExamplesChapter9EclipseStitchsrcStitch.g into the Stitch.g file you just created. The code you pasted is the ANTLR-based grammar that defines the syntax of the Stitch language. Listing 9-8 shows an abbreviated version of this code. A detailed explanation of the grammar is beyond the scope of this chapter, so I'll only explain the code briefly.

Stitch.g contains what is known as a context-free grammar. Context-free grammars are mathematically well-defined and explored, with the same level of expressiveness as the pushdown automata. If you want to learn more about the theoretical foundation of context-free grammars and pushdown automata, you can refer to books about automata theory or compiler construction. To learn more about ANTLR in particular, I recommend the books written by Terence Parr, the creator of ANTLR.

Before we look more closely at the code in Listing 9-8, let's briefly go over what a context-free grammar is made up of. It consists of production rules, each of which has a left-hand side and a right-hand side. In ANTLR's notation, a rule's left-hand side and right-hand side are separated by a colon and therefore look like this:

In this example, A is the left-hand side of a production rule and it's called a non-terminal. B C is the right-hand side, and B or C can be non-terminals or terminals. The difference between terminals and non-terminals is that terminals don't show up on the left-hand side of a production rule. Besides terminals and non-terminals, ANTLR allows us to have some C# code in a production rule. The C# code needs to be placed in curly braces, like this:

That's the one-minute introduction to context-free grammars and some of ANTLR's notations. There are many details that I omitted. I'll explain some of the details as we go over the code in Listing 9-8.

In Listing 9-8, line 1 declares the name of our grammar. Line 4 tells ANTLR to generate the lexer and parser code in C#. Lines 7 through 10 declare a @header block, which is the place where we can add extra C# using statements that the generated C# parser code will need. Everything in the @header block will show up in the generated parser code before the generated C# parser class. Line 12 tells ANTLR that we want the generated C# parser class to be in the Stitch namespace.

The real grammar of the Stitch language begins at line 14 and continues to the end of the file. The grammar basically consists of eight production rules. Listing 9-8 does not show the complete definition of each rule. We will see the complete definition of the program rule in the next few paragraphs. Because the rule definitions all follow the same pattern, after the program rule is explained in detail, I will go over the other rules only briefly.

The complete definition of the program rule in Stitch.g is like this:

If we strip out the C# code and some extra stuff (i.e. the returns declaration after program) that's mixed into the rule, the rule becomes this:

This much simpler form of the program rule basically says that the program non-terminal is made up of zero or more func (the asterisk means zero or more) non-terminals. Here the program non-terminal represents a Stitch script and the func non-terminal represents a Stitch function. We saw earlier in the “Stitch in Use” section that a Stitch script is made up of zero or more Stitch functions. So this rule is in line with the Stitch syntax we saw earlier.

Now let's look at the C# code and the returns declaration in the program rule. Both are in the production rule to serve the purpose of creating an abstract syntax tree (AST) from source code parsing. The C# code will obtain the result of the func rule and put it in an IList instance. If a Stitch script is made up of three Stitch functions, during source code parsing, the func rule will be applied three times and the IList instance that holds the results of the func rule will have three elements in it. We haven't explained the func rule yet but suffice it to say that the func rule returns an IFunction instance as the result. The IFunction interface is one of the Stitch AST classes defined in the C# Stitch project. Given the program rule, what happens when ANTLR generates the parser code is that ANTLR will generate a method whose name is also program in the generated C# parser class. The return type of the program method will be IList<IFunction>.

The program non-terminal is defined in terms of the func non-terminal. The func non-terminal is in turn defined in terms of some terminals and non-terminals. The non-terminals are defined in terms of other terminals or non-terminals and so on. If we strip out the C# code and the returns declaration that are mixed into each rule, like we did to the program rule, we get the simpler form of the rules as Table 9-1 shows.

Table 9-1. The Simpler Form of the Production Rules That Define the Syntax of the Stitch DSL.

Rule name	Rule definition
program rule	program : (func)* ;
func rule	func : '<' IDENT '(' parameters? ')' funcCode 'return(' parameters? ')>';
parameters rule	parameters : IDENT (',' IDENT)*;
funcCode rule	funcCode: (include | IDENT) CODEBLOCK;
include rule	include : 'include' FILEPATH;
CODEBLOCK rule	CODEBLOCK : '>' .* '<';
FILEPATH rule	FILEPATH : (LETTER | DIGIT) (LETTER | DIGIT | '.' | '' | '/')* '.' (LETTER | DIGIT)+;
IDENT rule	IDENT : LETTER (LETTER | DIGIT)*;

Test-Driving the Grammar

There are two ways to test drive the grammar defined in the last section. One way is to use the nice GUI feature that ANTLR IDE 2.1.0 provides. The other is to write some C# code that exercises the generated lexer and parser files. We will look at both approaches.

Figure 9-3 shows you how the GUI for testing a grammar looks in Eclipse. To try out the GUI feature, you need to first make Stitch.g the active file in the Eclipse IDE. Once Stitch.g is the active file, you'll see three tabs at the bottom of the code editor. The Grammar tab is for writing the grammar we saw in Listing 9-8. The Interpreter tab is for testing the grammar. After you select the Interpreter tab, you'll see a list of all parser and lexer rules defined in Stitch.g in the upper left area. The screen capture in Figure 9-3 shows that the rule “program” is selected. There are two panes that take up most of the screen in Figure 9-3. The upper pane shows the test script I typed in for testing the grammar. You can save the test script by pressing the Save icon in the upper right corner. To run the test script, you press the Run icon close to the Save icon. When you run a test script, you'll see the result as a tree in the lower pane. You can examine the tree to see if the grammar you defined parses the test script as expected.

Figure 9-3. Using the ANTLR IDE plug-in to debug a grammar

Another way to test a grammar is to write some code that exercises the generated lexer and parser code. Listing 9-9 shows the C# code that test-drives the StitchParser class and the StitchLexer class generated from Stitch.g. You'll find the code in Listing 9-9 in Program.cs of the StitchDemoApplication project.

This code example uses the file testScript1.st as the test script. The code in line 3 opens testScript1.st as a file stream. The file stream is passed to the lexer object in line 4. The code in line 5 creates a token stream out of the lexer object.  The token stream is passed to a parser object in line 6. And in line 7, the code calls the program method on the parser object. The program method represents the grammar rule program defined in Stitch.g. The program method returns a list of IFunction objects because in Stitch.g, the program rule is specified to return a list of IFunction objects as the result.

It's good to test-drive a grammar using the two techniques illustrated in this section and make sure that the grammar works as expected. Normally I use the GUI approach when I experiment with the grammar under creation. Once the grammar is stable, I use the C# approach to write unit tests that I can run to quickly and automatically check the correctness of the grammar.

Lexer and parser alone are not enough for executing Stitch code. The program method of the StitchParser class returns a list of IFunction objects. Each IFunction object represents a block of script code written in Python, Ruby, or some other language. We need to take those IFunction objects and figure out how to execute them. In the next section, we'll look at the Stitch runtime, which does exactly that.

The Stitch Runtime

The Stitch runtime is the component that executes Stitch code. This section and the next will give an overview of how the Stitch runtime performs its job. Subsequent sections will dive deeper into the details. Figure 9-4 shows the Stitch runtime (the Stitch box in the middle) in relation to the components it interacts with. The Stitch runtime interacts with two kinds of components—client applications and language plug-ins. For language plug-ins, the Stitch runtime provides two interfaces that serve as the contract for interactions. The two interfaces are ILanguagePlugin and IScript. If you want to extend the Stitch runtime by adding support for a new language, you need to implement those two interfaces. We will look at how to implement a Stitch language plug-in for Windows PowerShell later in the chapter.

Besides language plug-ins, the Stitch runtime interacts with client applications. A client application uses the Stitch runtime to run Stitch code, in one of two ways. One way is to use the StitchScriptEngine class in the C# Stitch project directly. The other way is to use the StitchScriptEngine indirectly via the DLR Hosting API. The Stitch runtime provides the StitchContext class and the StitchScriptCode class to support invoking the StitchScriptEngine via the DLR Hosting API.

Figure 9-4. The Script runtime in relation to the components it interacts with.

Overview of the Runtime

I'd like to give an overview of the Stitch runtime first. If you take the Stitch rectangle in Figure 9-4 and enlarge it, you'll see the subparts that make up the Stitch runtime and how they work together, as shown in Figure 9-5.  Figure 9-5 shows the flow of activities that take place when the Stitch runtime executes a Stitch script file. To execute Stitch code, the Stitch runtime uses a lexer to translate textual Stitch source code into a stream of tokens. The tokens are fed into a parser that turns the tokens into an abstract syntax tree, which is made up of instances of the classes in the Stitch.Ast namespace. We saw the lexer and parser in action when we looked at the code in Listing 9-9. The most interesting part of the abstract syntax tree is the list of IFunction objects returned by StitchParser's program method. The Stitch runtime uses a function execution coordinator to coordinate the execution of the list of IFunction objects. The Stitch runtime provides both a parallel coordinator and a sequential coordinator that you can use to coordinate the execution of Stitch functions.

At a very high level, a coordinator takes as input a list of IFunction objects and a registry of the languages supported by the Stitch runtime. For the Stitch runtime to support a language like PowerShell, you need to register the language's plug-in with the runtime. We will look at the language plug-in mechanism of Stitch in a later section. For now, let's focus on the overall flow of executing a Stitch script. Once a coordinator has the inputs it needs, it creates a script runner for each IFunction object. A script runner, as its name suggests, runs a block of script code. It does so by calling the Execute method of an IScript instance, which encapsulates the actual logic for executing a function. We will begin our exploration of the Stitch runtime by first looking at the script engine in the next section.

Figure 9-5. The flow of activities that take place when the Stitch runtime executes a Stitch script.

The Script Engine

We saw earlier how StitchScriptEngine is used to execute Stitch code. Using StitchScriptEngine to execute Stitch code is a two-step process. First you call StitchScriptEngine's constructor to create an instance of the class. Then you call either the RunScriptFile method or the RunScriptCode method to execute the Stitch code. When you call the StitchScriptEngine constructor, you need to pass in an execution mode and a collection of language plug-ins. The execution mode tells the Stitch runtime whether you want to execute Stitch code sequentially or in parallel. The collection of language plug-ins represents the languages you want to plug into the Stitch runtime. Listing 9-10 shows the code of the StitchScriptEngine's constructor. The important thing to note about the code in Listing 9-10 is that, depending on the execution mode, the StitchScriptEngine constructor will create either an instance of ParallelFunctionExecutionCoordinator or an instance of SequentialFunctionExecutionCoordinator. We will see an explanation of those coordinator classes in a minute.

StitchScriptEngine provides one method called RunScriptFile for executing Stitch code in a file and another method called RunScriptCode for executing Stitch code as a string. The implementations of the two methods are similar. Listing 9-11 shows the code inside the RunScriptCode method. This code is almost the same as the code in Listing 9-9. The only difference is that the RunScriptCode method takes the list of IFunction objects returned by the parser and passes it to the RunScripts method of a function execution coordinator, which is the topic of the next couple of sections.

Function Execution Coordinator

A Stitch script can have multiple Stitch functions. Some functions might depend on the output of other functions. Because of the dependencies among functions, the Stitch runtime uses a coordinator to manage the execution of functions. (This coordinator is illustrated in Figure 9-5.) The concept of function execution coordinators is defined as the IFunctionExecutionCoordinator interface in the C# Stitch project. Listing 9-12 shows the interface definition. The interface has only one method called RunScripts that takes as input a list of IFunction objects and a registry of languages supported by the Stitch runtime.

The C# Stitch project has two classes that implement the IFunctionExecutionCoordinator interface. Those two classes are ParallelFunctionExecutionCoordinator and SequentialFunctionExecutionCoordinator. ParallelFunctionExecutionCoordinator is used when we run Stitch code in parallel (i.e., when the execution mode is ExecutionMode.ParallelWaitAll or ExecutionMode.ParallelNoWait). SequentialFunctionExecutionCoordinator is used when we run Stitch code sequentially (i.e, when the execution mode is ExecutionMode.Sequential). When implementing the C# Stitch project, I thought about opening up the coordinator-related stuff so that new coordination logic could be plugged into the Stitch language runtime. For the sake of simplicity, I decided to leave that feature out of this chapter. The feature will be implemented in the dPier open source project at http://code.google.com/p/dpier/. In this section, we are going to look at only the ParallelFunctionExecutionCoordinator class. The implementation in the SequentialFunctionExecutionCoordinator class is much simpler and less interesting than ParallelFunctionExecutionCoordinator.

Listing 9-13 shows the RunScripts method implemented in ParallelFunctionExecutionCoordinator. The RunScripts method takes two parameters—functions and registry—as input. It calls the CreateScriptRunners method in the same class. The CreateScriptRunners method returns a ParallelScriptRunner instance for each IFunction object in the functions parameter. The idea of creating script runners here is that the coordinator class will concern only the coordination of executing multiple functions. It will not concern how each individual function is executed. The execution of a single, individual function is handled by a script runner, not by the coordinator. That's why the code in Listing 9-13 calls the CreateScriptRunners method to create a parallel script runner for each IFunction object. Once the script runners are created, the code in Listing 9-13 calls the Run method on those script runners to start the execution of Stitch functions.

Creating script runners for running Stitch functions in parallel requires some work. A parallel script runner can be created for running Python code, Ruby code, or another language's code. So, following good software design principles, the language-specific part (the part specific to how Python, Ruby, and other languages execute their code) of running a Stitch function is separated from the ParallelScriptRunner class and abstracted into the IScript interface. The IScript interface will be implemented by language plug-ins, which we'll see in a later section. With the IScript interface and Stitch's plug-in mechanism, we can use one single ParallelScriptRunner class to run different language code in parallel. ParallelScriptRunner keeps a reference to an IScript instance in its script field and delegates the execution of a Stitch function to that IScript instance.

When creating a parallel script runner for a Stitch function, we need to get an IScript instance that knows how to execute the language-specific code in a Stitch function. Listing 9-14 shows how to achieve that. Listing 9-14 shows the code in the CreateScriptRunners method of ParallelFunctionExecutionCoordinator. In line 10, the code calls the CreateScript method on the registry parameter to get an IScript instance that knows how to execute the language-specific code in a function.

Other than delegating the job of executing a Stitch function to an IScript instance, another main responsibility of the ParallelScriptRunner class is to keep track of a Stitch function's dependencies. If a Stitch function A depends on the return values of Stitch functions B and C, the parallel script runner for function A will keep in its prerequisites field a reference to each of B's and C's parallel script runners. That's why the code in line 26 calls the AddPrerequisite method on the scriptRunner object to track a Stitch function's dependencies.

Parallel Extensions for .NET

Because the parallel script runner we'll look at in the next section leverages the .NET Task Parallel Library (TPL) to do parallel programming, we will take a little detour and introduce the parts of TPL that are needed for our later discussion. TPL is a library within a larger component called Parallel Extensions. Before Parallel Extensions was released, I used to write code to deal with .NET thread pools. Thread pools in .NET are not friendly at all to developers. In those dark days, I lost a lot of brain cells, only to end up with code that was a headache to maintain. I can say from experience that TPL makes multithreaded programming much easier than thread pools do. Even though with TPL, writing parallel programs is simpler, it is still a very difficult thing to do correctly. That's why I'm attempting to make it easier to run scripts in parallel by implementing the Stitch language.

Listing 9-15 shows the TPL example we will look at in this section. The example creates in total eight TPL tasks. A TPL task is a unit of work that runs on a single thread. Multiple tasks can run on different threads in parallel if those tasks can be parallelized. One scenario in which two tasks can't be parallelized is when one task depends on the other. The eight TPL tasks in Listing 9-15 illustrate how task dependency affects parallel execution of tasks. In Listing 9-15, task1, task2, and task3 don't depend on anyone else. We create those tasks by calling the static Task.Factory.StartNew method and passing in a delegate that represents the actions we want the task to perform. Because we use the StartNew method to create those tasks, the tasks will start running after they are created.

task4 depends on task1. When we create task4, we use the static Task.Factory.ContinueWhenAll method. The first parameter to the ContinueWhenAll method is an array of tasks that task4 depends on. The second parameter is the delegate that represents the actions we want task4 to perform. Because we use the ContinueWhenAll method to create task4, task4 won't start executing until task1 is done. Similarly, task5 depends on task2; task6 on task4; task 7 on task2, task4 and task5; and finally task8 on task3 and task5. Every time you run the code in Listing 9-15, you will likely see “task 1”, “task 2”, “task 3”, and so on printed on the screen in different order. One thing you can count on is that “task 4” will never be printed before “task 1” because task4 won't start to run until task1 is done.

Script Runner

We will now use what we discussed about the Task Parallel Library to explain how the parallel script runner is implemented. A script runner is an object that knows how to run a script. As mentioned earlier, a script runner does not execute a Python script or Ruby script directly. Instead, it delegates that job to an IScript instance so that the script runner can remain language neutral. The concept of script runner is defined as the IScriptRunner interface in the C# Stitch project. Listing 9-16 shows the interface definition of IScriptRunner.

The IScriptRunner interface does not look very interesting. It has only a Run method that takes no input and returns no result. Let's see how the IScriptRunner interface is implemented by the ParallelScriptRunner class. The other class that implements the IScriptRunner interface is SequentialScriptRunner. SequentialScriptRunner is used by SequentialFunctionExecutionCoordinator whereas ParallelScriptRunner is used by ParallelFunctionExecutionCoordinator. Since the code in SequentialScriptRunner is relatively simple and straightforward, I will skip its explanation.

Listing 9-17 shows the StartTask method in the ParallelScriptRunner class. If a script runner does not have any prerequisites (line 6), then it can run immediately without waiting for other script runners to finish. So the code in line 8 calls the Task.Factory.StartNew method to start a new task immediately. The new task will run the lambda delegate that's passed to the TaskFactory.StartNew method call. The code in line 10 creates a dictionary object to hold the variables that are required by the language-specific code as input. Because the script runner does not have any prerequisites, the language-specific code does not require any input variables. That's why the dictionary object created in line 10 is an empty dictionary. The role the dictionary object plays is analogous to the role a ScriptScope instance plays in the DLR Hosting API. I chose to use a dictionary object as the carrier of the variables required by language-specific code because the Stitch language needs to support not only DLR-based languages but also non-DLR-based ones. So instead of using the DLR-specific ScriptScope class, I use IDictionary<String, Object>.

If a script runner has prerequisites, we need to run those prerequisites and wait for them to finish before we can kick off the script runner. The code in lines 19 and 20 loops through a script runner's prerequisites and starts them running. For each prerequisite, the code puts its task object (an instance of Task<IDictionary<String, Object>>) in the taskList variable. Then in line 22, the code calls the Task.Factory.ContinueWhenAll method to create a task object that will start to run only when all the tasks in taskList are finished. The task created in line 22 will run the lambda delegate defined in Listing 9-17 from line 23 to line 37. The lambda delegate prepares a dictionary object to hold the input variables needed by the language-specific code. This time, because the script runner has prerequisites, the dictionary object can't be empty. It needs to contain the results of the prerequisites' tasks. The code from line 25 to line 34 puts the results of the prerequisites' tasks into a dictionary object. Then, in line 36, the code calls the Execute method on the script variable, passing it the dictionary object to kick off the execution of language-specific code.

So far, we have looked at the function execution coordinator and script runner of the Stitch runtime. We saw how the coordinator coordinates and the runner runs. I mentioned that it's the IScript interface, not the IScriptRunner interface, that actually runs a Stitch function's language-specific code. The Stitch runtime comes with a built-in class DlrScript that implements the IScript interface for all DLR-based languages. For non-DLR-based languages like PowerShell, we can extend the Stitch runtime to support them by implementing the IScript interface and another interface called ILanguagePlugin. The two interfaces, IScript and ILanguagePlugin, make up the contract between the Stitch runtime and language plug-ins. Let's see the built-in DlrScript class in the next section. After that, we'll look at how the language plug-in mechanism works using the PowerShell plug-in as an example.

Running DLR-based Language Code

The IScript interface is meant to be implemented for each language that can be plugged into the Stitch runtime. Listing 9-18 shows the interface definition of IScript.

It turns out that for DLR-based languages like IronPython and IronRuby, because the DLR Hosting API provides a uniform way for executing those languages' code, we only need one implementation of the IScript interface for plugging all those languages into the Stitch runtime. The Stitch runtime comes with a built-in class, DlrScript, that implements the IScript interface for all DLR-based languages. Listing 9-19 shows the Execute method in DlrScript.

The Execute method takes a dictionary object as input. As explained in the previous section, the dictionary object carries the variables that are required by the language-specific code. The code in Listing 9-19 first creates an instance of ScriptScope and then copies the variables in the dictionary object to the ScriptScope object. Once the ScriptScope instance is ready, the Execute method uses the DLR Hosting API to get a ScriptEngine instance. The Execute method then uses the ScriptEngine instance to run the DLR-based language code. The results of this are in the ScriptScope instance the Execute method created earlier. Those results are copied into a dictionary object and returned to the caller of the Execute method.

Creating a new instance of ScriptScope in the Execute method means that we are not sharing a single ScriptScope object among the execution of multiple Stitch functions. This is important because Stitch functions can run in parallel and the ScriptScope class is not thread safe. We would have a lot of locking and thread synchronization to worry about if we shared a ScriptScope object in the concurrent execution of multiple Stitch functions.

One important thing to note about the code in Listing 9-19 is that the copying of variables from the dictionary object to the ScriptScope object is a shallow copying, meaning that the copying only copies object references and doesn't create new instances of those variables. That means even though the ScriptScope object is not shared among multiple Stitch functions, the variables contained in the ScriptScope object might be shared. And that seems to put us back to the thread synchronization issue we wanted to avoid. The reason I didn't implement a deep copying of the variables in the dictionary is because I expect the variables to be read-only. The design of the Stitch language adopts the functional programming paradigm and requires that if we intend to run Stitch functions in parallel, those Stitch functions should only produce new results and not alter their input variables. If all Stitch functions in a Stitch script don't alter their input variables, those variables are read-only and therefore are safe to be shared among multiple threads.

Language Plug-In

The last section shows the built-in support for executing DLR-based languages in the Stitch runtime. The Stitch runtime can be extended to support non-DLR-based languages if we provide Stitch plug-ins for those languages. The next few of sections will show how to develop a Stitch plug-in for the PowerShell language and how the plug-in mechanism works.

Develop a Stitch Plug-In for PowerShell

Developing a Stitch plug-in means implementing two interfaces: ILanguagePlugin and IScript. In this section, we are going to look at the implementation of those two interfaces for the PowerShell language. The implementation of the IScript interface will contain the logic for running PowerShell code. The implementation of the ILanguagePlugin will contain some information about the PowerShell language needed by the Stitch runtime. You can find those implementations in the PowerShellStitchPlugin project in this chapter's code download.

We have seen the IScript interface in the previous section. Listing 9-20 shows the code of the PowerShellScript class that implements the IScript interface for the PowerShell language. The logic for running PowerShell code is in the Execute method. I won't go into the details of how PowerShell works. For our purpose, it's enough to know that PowerShell uses something called a pipeline to execute PowerShell commands (i.e., PowerShell code) in a Runspace instance. The Runspace instance provides an execution context for the pipeline. The Execute method shown in Listing 9-20 creates a Runspace instance and a pipeline, puts PowerShell code as commands into the pipeline, and calls the pipeline's Invoke method to execute PowerShell code. The results returned by Invoke are the results of running PowerShell code. Those results are put into a dictionary object and returned to the caller of the Execute method. The code in the Execute method is for demonstration purposes. It does not use the Execute method's input parameter to set the input of the pipeline. That means the PowerShell code we write or include in a Stitch script can't take any input variables.

The other interface we need to implement for supporting PowerShell in the Stitch runtime is ILanguagePlugin. Listing 9-21 shows the interface definition of ILanguagePlugin, which defines a FileExtensions property for returning the file extensions of a language as a list of String objects. It also defines a LanguageNames property for returning the names of a language. When a language plug-in is hooked up (i.e., registered) with the Stitch runtime, the Stitch runtime will record the file extensions and language names so that the runtime can look up the right language plug-in by file extension or language name later.

For example, say we have the following Stitch code:

The language name in this case is “PowerShell.” When the Stitch runtime executes the code, it parses it into an IFunction instance. The IFunction instance has a String property called LanguageName, whose value in this case is “PowerShell”. The Stitch runtime queries a registry that contains information about language plug-ins. The registry keeps one dictionary that maps a language name to the corresponding ILanguagePlugin instance and another dictionary that maps a language's file extension to the corresponding ILanguagePlugin instance. The registry is an instance of the ILanguageRegistry interface, which is shown in Listing 9-22. The ILanguageRegistry interface defines a Register method for registering new language plug-ins and a CreateScript method that creates an IScript object for an IFunction object. The Stitch runtime queries a registry by calling the registry's CreateScript method and passing it an IFunction object. The CreateScript method of ILanguageRegistry, in our example, will get the “PowerShell” language name from the IFunction object and use that name to look up the corresponding language plug-in. The CreateScript method of ILanguageRegistry will then call the CreateScript method of ILanguagePlugin on the looked-up language plug-in. The IScript object returned by the CreateScript method of ILanguagePlugin will eventually be wrapped with either a parallel script runner or a sequential script runner so that the PowerShell code can run in parallel or in sequence with other Stitch functions.

Now that we've seen the ILanguagePlugin interface, let's see the PowerShellPlugin class that implements the ILanguagePlugin interface for the PowerShell language in Listing 9-23. The implementation is fairly straightforward. For ILanguagePlugin's FileExtensions property, the code simply returns “.sp”. For the LanguageNames property, the code returns “PowerShell”. The implementation of ILanguagePlugin's CreateScript method in Listing 9-23 simply creates and returns an instance of the PowerShellScript class we saw in Listing 9-20.

Configuring a Plug-In

Now that we have the PowerShell plug-in in hand, we need to register it with the Stitch runtime in order to use it. There are two ways to achieve this. One way is to pass an instance of PowerShellPlugin to the constructor of StitchScriptEngine as in the following code snippet:

The other way to register a language plug-in with the Stitch runtime is by configuration. Listing 9-24 shows what the configuration looks like in the App.config file. The configuration is based on the fact that Stitch is integrated with the DLR Hosting API. Because of this, we have the code from lines 5 through 8 to let the DLR Hosting API know about the Stitch language. And the code from lines 11 to 16 are the way the DLR Hosting API provides for passing custom information in the form of key-value pairs to a DLR Hosting API-enabled language. In this case, the custom information passed to the Stitch language runtime is two key-value pairs. The first pair has “plugin” as the key and the assembly-qualified type name of the PowerShellPlugin class as the value. This key-value pair will cause the PowerShellPlugin to be registered with the Stitch runtime. The second key-value pair has “executionMode” as the key and “ParallelNoWait” as the value. This key-value pair declares which execution mode we want the Stitch runtime to operate in.

Summary

In this chapter, we started with a list of abstractions that could be built on top the DLR Hosting API to make language hosting more declarative and less tied to the underlying .NET and DLR platform. We then went through the uses and implementations of the Stitch domain specific language that provide those abstractions. The Stitch language is declarative and platform-independent, and it can run both DLR-based and non-DLR based-language code in sequence or in parallel. We looked at the design and grammar definition of Stitch's syntax, and at the key components, such as the function execution coordinator and script runner, in the Stitch runtime. Finally, we saw how to implement a Stitch plug-in for the PowerShell language, and how to register the plug-in with the Stitch runtime.

There are many features and improvements that could be added to the Stitch language as presented in this chapter. For example, we could implement language plug-ins for running Ant, Maven, or other language scripts. The Stitch language can make the execution modes extensible and allow language scripts to run in different ways. I created the dpier project up on Google Code at http://code.google.com/p/dpier/ for the continuous development of the Stitch language. You can visit the project web site to get the latest updates and give feedback.