P A R T  1

DLR Fundamentals

C H A P T E R  1

Introduction to DLR

DLR stands for dynamic language runtime. Maybe you already know something about it and the reason you picked up this book is to learn how the DLR works and how to make use of it. If you haven't heard of the DLR, you may be wondering whether it's worth your time learning it. One reason people might regard the DLR as irrelevant to their work is that they think the DLR is for implementing new languages. And since most of us write programs to solve specific problems and very few of us implement languages, learning the DLR may not seem like a good investment. That was in fact my initial misconception when I first heard of the DLR, around the time it was announced in 2007. After some study, I quickly realized the broad applicability of the DLR in many areas of my day-to-day programming work.

Because of that potential misconception, I want to highlight some areas in which the DLR shines. The point I want to get across is that the DLR is not merely for running or implementing dynamic languages. It is also very useful for application scripting, meta-programming, aspect-oriented programming (AOP), building DSLs (domain-specific languages), unit test mocking, and a lot more. Instead of just throwing out those buzz words and iterating through them with dry discussions, I figure the best way to highlight the practical usefulness of the DLR is through some examples. So that's what this chapter will do. Normally an introductory chapter like this has a Hello World example. We will have not just one, but four, plus some demonstrations.

Since most people know the DLR as a platform for building and running dynamic languages, we'll start with a Hello World example of running a dynamic language. Next we'll show a Hello World example of building a dynamic language. We'll then take that language and show how to embed it in a host application written in C#. Finally, we'll end the series of Hello World examples with a REPL (read-eval-print-loop) console for the Hello World language. It might seem strange to use the building of a programming language as a Hello World example. After all, building a programming language is no trivial task. But, as you will see, because of the rich features the DLR provides, we can do all the things mentioned with very little code.

The series of Hello World examples is about using, embedding, and building programming languages. But the DLR also does a good job of enabling application scripting. The DLR makes it very easy to add scripting capacity to your applications. Users of your application can take advantage of popular dynamic languages such as IronPython and IronRuby to extend your application with custom capabilities, automate certain tasks, or integrate your application with other systems. If you like, you can choose to create your own domain-specific language and let users script in a syntax closer to the domain of your application. And that's what later chapters in this book will show you. In Chapter 10, you'll implement a fun WPF application that uses a physics engine to detect collisions between balls. You can write IronPython code to script the movement of balls. You can also write code in the DSL that Chapter 12 will explore, to do things like stopping and starting a ball. I'll give a preview of that WPF application in this chapter.

After that, we'll see how the DLR makes it very intuitive to work with data. The technique we'll use is often referred to as builders. In Chapter 5, we'll explore the XML builder library we'll use to build XML data.

Finally, we'll delve into the Aspect-Oriented Programming (AOP) framework covered in Chapter 7. AOP is a programming paradigm that is very well-suited to solving the problem of cross-cutting concerns. Common cross-cutting concerns in a software system are things like transaction management, security, auditing, performance monitoring, logging, and tracing, and the like. By virtue of addressing the problem of cross-cutting concerns in an elegant manner, AOP provides tremendous value in the design and architecture of software systems. As you'll see, one nice thing about the AOP framework is that it works across dynamic and static languages, and it's also integrated with Spring.NET's AOP framework.

Even though this chapter will not get into the details of the demonstrations, I hope after seeing the examples and demonstrations, you'll feel that the benefits and applicability of DLR advocated here are more real and tangible. Without further ado, let's begin by setting up the software components needed for running the examples.

Setting Up Code Examples

If you download and unzip the file that contains this book's code examples, you'll see the following file structure:

The Examples folder contains a subfolder for each chapter where you can find all of a chapter's code examples. Most of the code examples depend on one or more software components. The lib folder has a subfolder for each of the software components used in this book. You'll need to download those components and put the needed assembly files into the subfolders under the lib folder. The next section will describe what you need to do to download the DLR assemblies and put them into the libDLR folder. For the other software components, I'll describe how to set them up when we encounter them in later chapters. Throughout the book, I'll assume that the ProDLR folder is placed under C:. If you choose to place it in a different folder, you'll need to substitute the path with your own whenever I refer to it in the book.

Software Requirements

For most of the examples in this book, you'll need the following software to follow along:

• .NET 4.0 SDK: You can download this from Microsoft's web site and follow the instructions there to install it.
• Visual Studio 2010 Express: Although you technically don't need to install this, it is highly recommended as it will make following the code examples much easier. The installation of Visual Studio 2010 Express also installs the .NET 4.0 SDK, so if you choose to install this component, you don't need to install the .NET 4.0 SDK separately.
• DLR, IronPython, and IronRuby: You can go to the DLR project web site at CodePlex to download all three in one bundle. At the time of this writing, the download page of the DLR CodePlex website provides only source code, no binaries. The next section will describe where to get the binaries and how to install them.

The DLR, IronPython, and IronRuby can run on .NET 2.0. To do so, you'll need to download different binaries from the IronPython and IronRuby websites. As we go through the installation of the DLR, IronPython, and IronRuby in the next section, I'll point out the binaries you need if you want to use .NET 2.0 as the target platform. The code examples in this book are developed to run on .NET 4.0, but Chapter 3 shows you how to target both .NET 2.0 and .NET 4.0.      

Installing the DLR, IronPython, and IronRuby

Even though the files you download from the DLR CodePlex web site contain only the source code, you can get the DLR binaries from IronPython's or IronRuby's CodePlex web sites. Here are the steps you need to take to get the release bits of DLR, IronPython, and IronRuby.

1. Go to ironpython.codeplex.com/ and download IronPython 2.6.1 for .NET 4.0. That's the version I use for this book's code examples; it's an .msi file. You simply double-click it and follow the instructions to install IronPython. From now on, I'll assume that it's installed in C:Program Files (x86)IronPython 2.6 for .NET 4.0. If you choose to install it in a different folder, you'll need to substitute the path with your own whenever I refer to it in the book. If you need to develop DLR-based applications that run on .NET 2.0, download IronPython 2.6.1 for .NET 2.0 SP1 instead.
2. Go to http://ironruby.codeplex.com/ and download IronRuby 1.0 for .NET 4.0 (ironruby-1.0v4.msi). That is the version of IronRuby I use in this book. Again simply double-click on it and follow the instruction to install it. I'll assume that it's installed in C:Program Files (x86)IronRuby 1.0v4. If you need to develop DLR-based applications that run on .NET 2.0, download IronRuby 1.0 for .NET 2.0 SP1 instead.
3. Copy the following files from C:Program Files (x86)IronRuby 1.0v4in to C:ProDLRlibDLR elease:
   • IronRuby.dll
   • IronRuby.Libraries.dll
   • Microsoft.Dynamic.dll (This and the next assembly are the DLR version 1.0 binaries).
   • Microsoft.Scripting.dll
4. Copy the following files from C:Program Files (x86)IronPython 2.6 for .NET 4.0 to C:ProDLRlibDLR elease:
   • IronPython.dll
   • IronPython.Modules.dll

To make it convenient to run the Read-Eval-Print-Loop (REPL) consoles of IronPython and IronRuby, you might want to make sure that "C:Program Files (x86)IronPython 2.6 for .NET 4.0" and "C:Program Files (x86)IronRuby 1.0v4in" are included in your Path environment variable.

Hello World Examples

Now let's get started with our four Hello World examples that run, build, and embed DLR-based dynamic languages. Let's have some fun!

Hello World from a Dynamic Language

We'll first look at an implementation of a Hello World program in a dynamic language. IronPython is one of the most mature DLR-based dynamic languages. There are other implementations of the Python language, such as CPython and Jython. But those are based on their own runtimes, not on DLR. Figure 1-1 shows IronPython code that prints “Hello World” to IronPython's interactive console.

Figure 1-1. Hello World from IronPython

That looks fairly straightforward. Like most dynamic languages, IronPython comes with an interactive console that reads the code you type in, evaluates that code, prints out the result of the evaluation, and waits for the next code snippet. This is commonly called a REPL (Read-Eval-Print-Loop) console. IronPython's REPL console is the ipy.exe executable file. If you've put the right paths in the Windows Path environment variable as mentioned earlier, you should be able to just type in the command console to execute ipy.exe. When prompted for input by >>>, you type the code print “Hello World”, and the result of evaluating that code is printed in the next line.

Creating a “Hello” Language

Next, we'll look at implementing our own language. Yes, we'll create a whole, new language just for “Hello World.” We will implement a DLR-based language that we'll call the Hello language, and it will print “Hello World” to the console no matter what Hello code you write. In other words, the language accepts any input code as valid Hello code and responds to that input code by printing out “Hello World”. Because any Hello code is valid, we don't need a parser to parse the code. Nor do we need to interpret or compile any code. You can find the source code for this example is in the HelloLanguage project of the Chapter 1 solution.

Implementing a DLR-based language as simple as the Hello language means we need to implement two things: a language context and a script code. I'll explain language context and script code in more detail when we get to the discussion of the DLR Hosting API in Chapter 6. For the time being, we can understand language context as something that provides an entry point to a language's compilation capability. As for script code, just think of it as a representation of a language's compiled code for now.

Listing 1-1 shows the language context implementation for the Hello language. All of the code is boilerplate except the line in bold. We can see that a language context provides an entry point method CompileSourceCode that invokes the source code compilation capability of a language. The source code is represented as an instance of type SourceUnit and passed as an input argument to the CompileSourceCode method. Our implementation of the CompileSourceCode method simply creates an instance of HelloScriptCode, which represents the result of compiling the input source code, i.e., the SourceUnit instance.

Listing 1-2 shows the code for the HelloScriptCode class. HelloScriptCode has a method Run, which is supposed to know how to run Hello code. Since running Hello code means printing “Hello World” to the console, we do just that in the Run method and that's all. We have completed our first DLR-based language.

Embedding the Hello Language

Now that we've built a language, let's put it into use. In this example, we'll embed the Hello language in a host application written in C#. In this case, C# is said to be the host language. To achieve the goal, we need to write the C# host application, of course, and to set up proper configurations in the App.config file so the C# host application can locate the Hello language.

Listing 1-3 shows the configurations you need to put into App.config. Don't worry too much if you don't fully understand every line of the code shown here. We'll delve into it more when we discuss the DLR Hosting API in Chapter 6. Essentially, the configurations tell the DLR runtime that there is a language called Hello and that its language context is the type HelloLanguage.HelloContext we saw in the previous example.

With the App.config file ready, there isn't much left to do. Listing 1-4 shows the C# code for the application that hosts the Hello language. Line 3 in Listing 1-4 reads the configuration information from App.config and creates an instance of the ScriptRuntime class. Given the information in App.config, line 4 is able to call our Hello language context behind the scenes and return an instance of ScriptEngine. The code in line 5 passes in “any text” as the input Hello code to the script engine. Line 6 calls the Execute method of ScriptSource to execute the Hello code and the words “Hello World” are printed to the console.

Implementing REPL for the Hello Language

With very little effort, we can provide a REPL console for the Hello language like IronPython's. For this task, we'll leverage the ConsoleHost class that the DLR provides. Listing 1-5 shows how to use this class to implement the console.

As Listing 1-5 shows, all you need to do is override the Provider property and make it return the type HelloLanguage.HelloContext. The code is so simple because the class does most of the heavy lifting, like printing the >>> prompt, reading user input, and calling HelloContext to execute that input. The REPL console in this example does not run by itself. We need the code in Listing 1-6 to run it.

Figure 1-2 shows the result of running the program. As you can see, it doesn't matter what input code we type, the Hello REPL console always prints “Hello” to the screen. To exit the REPL, press Ctrl+Z.

Figure 1-2. Running the Hello language's REPL console

Practical Uses for the DLR

The next few subsections show some of the code examples you'll develop as you follow along in this book. The main purpose of these demonstrations is to get across the point that the DLR is more broadly applicable than for just implementing and running dynamic languages. Furthermore, I hope the demonstrations will whet your appetite and make you look forward to the later chapters where we'll explain the source code in detail. I'll indicate for each demonstration the chapter it belongs to.

Application Scripting and DSL

Figure 1-3 shows a screen capture of the WPF application we'll develop in chapter 10. The application uses the Farseer Physics Engine to detect collisions between moving balls. The Farseer Physics Engine is a project hosted on the CodePlex site. It's a 2D physics engine that is very fun to work with. It's even more fun when we introduce scripting and extensibility into the application we build. For example, we can implement a domain-specific language (DSL) with commands such as Start Ball or Stop Ball. Users of the application will be able to write some script code to control ball movements.

Figure 1-3. A scriptable WPF application that does collision dectection

XML Builder

DLR makes it possible to design fluent APIs for working with certain kinds of data. For example, the “static” classes like XmlDocument, XmlElement, and  so forth in the .NET Class Library have methods for getting an XML element or attribute by name. With the DLR, we can implement a more fluent API whose method names are directly the names of the XML elements we want to access. Listing 1-7 shows what that looks like when we use a dynamic library to build an XML snippet.

The output of running the code in Listing 1-7 is shown below. As you can see in Listing 1-7, instead of calling methods that access XML elements by name, we access those names directly as if they are methods. Of course, the library works with any XML data you might want to build. I didn't implement the library in such a way that it recognizes only the Customer, Address, and other elements shown in the example. That would be cheating.

The DLR is not the only component that allows the design of fluent APIs like the one shown in the example. The Groovy language has this capability too and people have implemented many builders in Groovy for working with different kinds of data. Besides XML, the builder technique can also be applied to data sources such as file systems, registry entries, IT management, and more. For example, instead of a library that accesses files and folders in a file system by name, we can imagine a dynamic, fluent library that accesses files and folders as if their names are methods. Instead of accessing registry entries by name, we can access them as if those names are methods. You see the pattern.

Aspect-Oriented Programming

Aspect-oriented programming (AOP) is a programming paradigm for solving the problem of cross-cutting concerns, such as transaction management, auditing, security, logging, and the like. There are several techniques for implementing an AOP system. Some implementations use compile-time code weaving. Some use load-time code weaving. Some use runtime method interception. And most use a combination of these approaches. In Chapter 7, we'll see how to implement a DLR-based AOP framework that works across both static and dynamic languages. The AOP approach used in that chapter is runtime method interception. I will have more to say about the different AOP approaches when we get to Chapter 7. Here I'll just demonstrate what it looks like when we use the AOP framework to log the activities that take place in both a static object and a dynamic object.

Listing 1-8 contains two objects, customer and employee. The employee object is a typical static object that we are all familiar with. Line 7 obtains the employee object from the Spring.NET container (because our AOP framework is integrated with Spring.NET's AOP framework). The other object, customer, is a dynamic object created in line 3.

Figure 1-4 shows the result of running the program in Listing 1-8. As you can see, for both the customer and employee objects, the same AOP logging logic is applied and the console shows log statements for both objects.

Figure 1-4. Running Listing 1-8 produces logging for both a dynamic and a static object.

Now that you've seen the Hello World examples and some practical applications of the DLR, I will introduce the DLR itself and describe what it is. Of course, the name Dynamic Language Runtime conveys a lot about what DLR is. So let's kick off the discussion by looking at each of the words that make up the name, starting with runtime.

Runtime

The DLR is a runtime—that's pretty obvious. A runtime is a software component that does its work at the time a program runs. Let's say you use the .NET SDK in the development of your program. The SDK comes with both runtime components and development tools. You use the development tools like the C# compiler to develop and build your application. When you ship your application to customers, you don't need to include those tools in your distribution. In contrast, you do need to include the runtime components of the SDK because your application depends on them to function properly.

So the fact that the DLR is a runtime means that if we build our applications on top of the DLR, we need to distribute it together with our applications. More specifically, it means we will need to redistribute assemblies such as Microsoft.Scripting.dlland Microsoft.Dynamic.dll, two of the assemblies that make up the DLR.

Different runtime components do different kinds of work. One major kind of work the DLR does is executing expressions, and that's something we'll explore in great length in the next chapter. For now, you can think of expressions as a kind of intermediate language like MSIL (Microsoft Intermediate Language). Expressions are to the DLR what MSIL is to the CLR (Common Language Runtime). There are many other runtimes that execute code. Just as the CLR executes MSIL code compiled from C# or VB.NET programs, the JVM (Java Virtual Machine) executes Java byte code, and IronPython runtime interprets Python code.

The DLR executes expressions by interpreting them or by compiling them into MSIL. If the DLR compiles expressions into MSIL, the MSIL code will be sent to the CLR for execution. Figure 1-5 shows this flow: The tree at the left is fed into the DLR's expression compiler. Out of the compiler comes some MSIL code, which is fed into the CLR for execution. As you can see, DLR is built on top of CLR and leverages CLR to execute the MSIL code it generates.  

Figure 1-5. How the DLR executes expressions

As a runtime, the CLR provides services, such as garbage collection, runtime type checking, code access security, etc. Like the CLR, the DLR provides many services, including:

• Hosting API
• Debugging API
• Call site caching
• Expression compilation and interpretation
• Meta-object protocol for dynamic language interoperability

Besides the services listed above, because the DLR is built on top of the CLR, applications based on the DLR automatically benefit from the CLR's services. For example, .NET Application Domain and Remoting are leveraged in the DLR Hosting API to allow hosting dynamic languages in separate process or on a remote machine. The CLR's garbage collection is readily available to dynamic languages built on the DLR. If you were going to implement a language without taking advantage of the DLR, CLR, or JVM for that matter, imagine how much work you'd have to do to, for example, implement a garbage collector for your language's runtime.

While we are on the subject of runtimes and how they (i.e., the CLR and DLR) can be built on one another, let's throw IronPython into the picture. As mentioned earlier, IronPython is a runtime that interprets Python code. It is built on DLR. So when we run IronPython code, there will be three language runtimes, CLR, DLR, and IronPython, working together one on top of another.

Runtime vs. Run Time

To avoid confusion, I'd like to make clear the distinction between the word runtime and the phrase  “run time.” Runtime in our context is a software component. Run time, on the other hand, means the time when our code runs. If you say “my program throws an exception at run time,” people will understand that as “your program throws an exception when it runs.” However, it would be really odd if you say “my program throws an exception at runtime.” The word runtime and the phrase ‘run time' might appear to be interchangeable in other books or articles. The distinction we make here is only for the purpose of our discussions throughout the book.

Run Time vs. Compile Time

The terms  “run time” and “compile time” (or compile-time) are often used to indicate whether something occurs when a program is executed, or when it is compiled. Run time is usually contrasted with compile-time. Run time means the time when the code runs. Compile time means the time when code is being compiled. In this section, we will compare run time and compile time by looking at the typical flow of activities of each.

Figure 1-6 shows the typical flow of compile-time activities. Basically, at compile time the compiler takes source code as input and parses it. If the code does not conform to the syntax rules of the language, the compiler stops and reports errors. If the code passes the parsing phase, the compiler typically creates an intermediate representation of the source code in the form of a tree data structure. The tree data structure is often called an AST (abstract syntax tree). The compiler uses the AST to perform code analysis, such as type checking and possibly some code optimizations. In the end, the compiler generates the output binaries. Different compilers generate different binary files. C# compiler generates .NET assemblies; Java compiler, JVM byte code; and C++ compiler, machine code.

Figure 1-6. Typical flow of compile-time activities

Run time flow basically picks up where the compile-time flow left off. It takes the binaries generated as the final output of compilation and executes them. Figure 1-7 shows the typical flow of run-time activities. The flow begins with the runtime loading the binaries into memory. If the binaries are compiled from C++ code, they will contain machine code and the C++ runtime will directly execute the machine code. If the binaries are compiled from C# or VB.NET, the binaries will contain .NET IL and the CLR will at its discretion interpret the IL or JIT (just-in-time) compile the IL to machine code.  

Figure 1-7. Typical flow of run-time activities

So far, I have described the flow of run-time and compile-time activities in general. Let's look at them again, this time specific to DLR-based languages and to the DLR itself. We will look at DLR-based languages first and then at the DLR itself.

A DLR-based language may or may not have a compiler. Even if it has one, you may have the choice of not using it. For example, IronPython provides a compiler you can use to compile IronPython code into IL. But depending on the situation, you might not always want to do the compilation. For example, in a scenario where you provide scripting capability and let users write IronPython code to automate or extend your application, since the IronPython code is written by users, you probably want to make it easier by not requiring them to compile their code. The code snippet below shows the IronPython code that compiles the source file sample.py into the .NET assembly file sample.dll.

Normally a DLR-based language like IronPython parses input source code, builds its own AST, translates that AST to DLR's AST, and invokes the DLR to interpret or compile the DLR AST. Figure 1-8 shows the flow of compiling Python code into MSIL.

Figure 1-8. IronPython source file compilation flow

The DLR AST shown in the figure is nothing new, just the DLR expressions mentioned earlier. They are implemented as classes in the System.Linq.Expressions namespace inside the System.Core.dll assembly. The DLR AST in Figure 1-8 serves as input to the DLR expression compiler, which is implemented in the class Microsoft.Scripting.Ast.LambdaCompiler. Compiling expressions is not an easy task. Instead of doing all the work of compiling expressions by itself, LambdaCompiler leverages classes in the Microsoft.Scripting.Generation namespace to compile expressions into MSIL code. Those classes in turn use classes in the System.Reflection.Emit namespace, which you'll be familiar with if you've done some work at the MSIL level. The DLR, at its discretion, can also decide to interpret expressions instead of compiling them.

Dynamic vs. Static

We looked at the word runtime in the previous section. Let's look at another word that's part of DLR's name—dynamic. The word “dynamic” is in contrast to the word “static.” In our context, we are talking about dynamic and static languages. One fundamental question we need to answer is, how do we decide which languages are dynamic languages and which are static? On the surface, it might seem that a language is dynamic if you don't need to specify types for things like function arguments and variables when you write code in that language. That is, however, not true, and a counter example is F#. F# does not require you to provide type information in your code, yet it's a static language. It enforces type rules by performing type inference and type checking at compile time.

Another criterion that might seem like a good indicator is whether the language has a compiler. As it turns out, this is true in many but not all cases. We saw a counter example of this in the previous section. There we mentioned that you can compile IronPython code into IL—and IronPython is a dynamic language.

In fact, whether a language is dynamic or static is a matter of degree. Languages that we generally regard as static, such as C# and Java, are not entirely static. I'll give an example of what I mean by that shortly. Let's explain what we mean by dynamic and static first. In the context of programming languages, being dynamic means doing things at run time. In contrast, being static means doing things at compile time. With that in mind, an example of C# or Java being dynamic is array bounds checking. The C# compiler by design does not check and catch the following error:

The C# compiler will give green light to the code above and we will get an array index out-of-bounds exception at run time. As you can see, the checking of array index bounds is not performed statically at compile time, but dynamically at run time.

So why doesn't the C# compiler catch the error? After all, isn't it pretty obvious that the array contains only two items? While the array index out-of- bounds error is obvious in the code snippet above, that same error (if it exists) might not be obvious in this code:

The C# compiler simply can't know how many names there are in the database unless it runs the code. The array bounds checking is a classic example of activities that even static languages would perform at run time. Moreover, there's the new “dynamic” keyword that's been added to C# 4.0. With that new feature, it becomes even clearer that there's often no black and white divide when it comes to deciding whether a language is dynamic.

If we accept that whether a language is dynamic or static is a matter of degree, here are two key factors people often use to determine that degree:

• dynamic typing
• dynamic dispatch (aka late binding)

If a language exhibits some or all of both, it's often regarded as being more dynamic. We will go over these items in the next sections. In each section, we will describe the subject in general and also specifically as it relates to the DLR.

Dynamic Typing

Both static and dynamic languages can be strongly typed. The main difference is when they do type checking. Strongly typed static languages perform type checking at compile time while strongly typed dynamic languages perform type checking at run time. It would be much clearer if we called them dynamically checked and statically checked languages. But the terms static and dynamic languages are already in wide use.

Besides that main difference, there are two other key type-related differences between static and dynamic languages: changing types of variables and changing the definition of types. Let's look at some examples.

The C# code below causes a compilation error because C#, like most static languages, does not allow you to change the type of a variable.

The following C# code again causes a compilation error, even though we use the var keyword to declare the type of variable i. The difference between this code and the previous is that the previous code snippet explicitly tells the C# compiler that the type of variable i is int whereas this code tells the C# compiler to infer the type of variable i for us. Sure enough the C# compiler is able to do the inference and it dutifully reports back a compilation error to us.

In C# 4.0, there is a new language keyword called dynamic. If we use that keyword to declare the type of a variable, we are telling the C# compiler that the variable's type can change. The code snippet below shows an example. The variable i is not fixed to a single type. In other words, its type is dynamic, not static, and the C# compiler won't bother doing type checking on the variable i at compile time. If we compare this example to the previous two examples, the difference should be clear. Note that all three of these code snippets are strongly typed. The first two are strongly typed and their type correctness is checked at compile time. The last code snippet is also strongly typed and its type correctness is checked at run time.

C# allows variables to take on different types. Dynamic languages do that too, and more. In dynamic languages, it's not only variables that can take on different types; statements like if and switch can have different types as well. The Python example below shows an if statement that returns a number in its if-branch and a string in its else-branch. The if statement resides in a function called callIf. The function is called once with the integer 6, which causes the if-branch to be executed. The function is then called with the integer 4 to cause the else-branch to be executed. If you do something like this in C#, you'll get a compilation error because the C# language requires that you return objects of the same type in both branches.

The result of running this code looks like this:

The other major difference between static and dynamic languages is changing a type's definition at run time. Static languages typically don't allow that. With dynamic languages like Python and Ruby, we can change a type's definition by, say, adding new methods to it. Listing 1-9 shows some Ruby code that adds new methods to a type's definition at both the class level and instance level. The example first defines a class called Customer. The Customer class has an initialize method and a callRep method. The initialize method is a special method in Ruby. It is called after an instance of the class is created to initialize the newly created instance. In this example, the Customer class has a member variable called name. The initialize method sets the member variable name to the value passed to it. The callRep method of the Customer class simply prints a message to the console indicating that the customer's representative is called.

Once the Customer class is defined, the example code in Listing 1-9 creates two customer objects: bill and bob. Here's where things get interesting. Notice that bob's callRep method is redefined to print a different message that indicates the customer has no representative assigned to him. This is an example of redefining a method at the instance level. It's quite common to be able to do that in dynamic languages, but not in static languages. Because the callRep method is redefined only for bob, bill is not affected.

Next, the code shows an example of modifying a class's definition. It does so by adding a new method called makeReferral to the Customer class. Because the new method is added to the Customer class, both bob and bill will have that method defined for them. Again, changing a class's definition at run time is quite common in dynamic languages, but not so in static languages.

To run the code in Listing 1-9, first save the code into a file called something like Customer.rb. Then open a command console, navigate to the folder where the file Customer.rb resides, and simply execute ir Customer.rb. In the command, “ir” refers to the REPL executable of IronRuby. And I'm assuming that "C:Program Files (x86)IronRuby 1.0v4in" is in your Path environment variable. When running the code in Listing 1-9, you should see output like the following:

Dynamic Dispatch

When we do coding, we generally write code that calls a method, creates a new instance of a class, applies an arithmetic operator on some operands, and so forth. A language compiler or interpreter needs to know what to do when it encounters that code. When a line of code is a method invocation, the compiler/interpreter needs to know which method of which class to invoke. The method might be overloaded and, if that's the case, the compiler/interpreter needs to resolve that and pick the right method based on the input arguments passed to the method invocation. Similarly, when the line of code is an application of an operator on some operands, the compiler/interpreter needs to know which operator it should use. The operator can be overloaded for operands of different types. If that's the case, then again some kind of resolution based on the operands' types is necessary. The resolutions we talk about here are also called method bindings or method dispatches. In static languages, the bindings are done at compile time and because compile time happens earlier than run time, the bindings are also called early bindings. In dynamic languages, the bindings are done at run time and therefore are often called late bindings. Another commonly used term that's interchangeable with late binding is dynamic dispatch. The C# code below is an example of early binding. At compile time, the C# compiler knows the type of the variable name is String. When we call the ToLower method on name, the compiler knows that we're calling the ToLower method of the String class. It will also check and make sure that the String class has a ToLower method that takes no input arguments and returns a String instance.

The IronPython code below is an example of late binding. Basically, all of the things the C# compiler does to the previous C# code snippet are now done at run time by the IronPython runtime.

You don't need to supply type information in the Python code and there is no type checking during development. But when you run the code, the IronPython runtime performs similar type checking and method binding that the C# compiler does to the C# code at compile time.

Language

The final word we haven't talked about that makes up the name of the DLR is language. A programming language—or a human language, for that matter—essentially consists of two parts: syntax and semantics. Syntax is the form. Semantics is the meaning of the form. For example, in English, the literal form of the sentence “Roses are red” is in the realm of syntax. Its semantics is the meaning we associate with it, the fact that roses are red.

Different programming languages define different forms of the if statement. The forms look similar, but they are not exactly the same. If you take one language's form of the if statement and use it in another language, that other language won't recognize that form and it will throw an error when it tries to parse the if statement. However, although if statements in different languages differ in their forms, they pretty much have the same semantic meaning. This is analogous to human languages. English, Chinese, and Spanish each have a syntactic form for writing down the sentence “Roses are red,” and those forms all have the same meaning.

This observation of syntax and semantics leads very well to what the DLR provides. Because syntactic forms usually vary from language to language, the DLR doesn't restrict you to any specific form when you design a DLR-based language. You are totally free in defining the syntax of your language and in parsing that syntax. You define a language's syntax by specifying its grammar rules. In English, the sentence “Roses are red” is syntactically valid because it obeys the grammar rules of English. The sentence “Roses are not flowers” is also grammatically correct. But if you say it, you'll get “run time exceptions” (people frowning at you) because the sentence does not make sense semantically.

Since different if statements in different programming languages have the same semantic meaning, it's not surprising that the DLR provides a common semantics model for all DLR-based languages. That common semantics model is DLR expressions. When a DLR-based language maps its syntactic forms to DLR expressions, it essentially defines the semantics of those syntactic forms. DLR expressions play a pivotal role in the overall DLR architecture. It's the component that stitches together all the other core DLR components and makes the whole larger than the sum of the parts. Understanding DLR expressions is crucial in understanding how DLR works, and we will delve into them in the next chapter.

DLR expressions are a superset of LINQ expressions. In terms of implementation, LINQ expressions are the classes in the System.Linq.Expressions namespace of the System.Core.dll assembly. The DLR adds some expression classes of its own on top of LINQ expressions. Those DLR expressions are packaged in the Microsoft.Scripting.Ast namespace of the Microsoft.Dynamic.dll assembly. Throughout this book, I will use the terms DLR expressions and LINQ expressions interchangeably.

Although I just said that DLR expressions offer a common semantics model for all DLR-based languages, you can view them and use them as a syntax model if you like. That might seem confusing at first, but it's in fact not as blurry as it seems. Expressions are just objects. If we use the DLR runtime to execute them, then they have semantics and those are the semantics defined by the DLR. However, if we don't use the DLR to execute the expressions and instead interpret the expressions ourselves, those expressions are just a form of syntax up to us as to how we want to interpret them.

The LINQ query-provider mechanism is a good example of using LINQ expressions as a syntax model. Figure 1-9 shows what we mean by that statement, using LINQ to SQL as an example.

Figure 1-9. Using expressions as a syntax model

In the figure, the overall query process begins with the LINQ query at the top. The query is to retrieve the names of all customers who live in New York. It is transformed into a LINQ expression tree whose meaning is still open for interpretation. When the expression tree crosses the dashed line and reaches the LINQ to SQL query provider, it will then have the semantic meaning the query provider gives it. The query provider will interpret the expression tree according to the semantic meaning it has for the tree and the result of that is a SQL database query that can be executed to retrieve customers who live in New York.

Programming Languages in Practice

I just said that a programming language is essentially syntax plus semantics. Well, that's essentially true, but it's practically not true. Practically, using a programming language is more than just knowing the language's syntax and semantics. More often than not, you'll need a nice code editor that gives help with syntax, a debugger that allows you to step in and out of the code, libraries so that you don't need to code everything from scratch, a unit testing framework for obvious reasons, and maybe an IDE that integrates all the different pieces together in a nice and intuitive way to improve your already remarkable productivity even further. In addition to all those, you might also want to have tools for doing static code analysis, unit test coverage, performance benchmarks, and more. Table 1-1 contrasts C# with typical DLR-based languages. The table shows what you get with C# compared with what you typically need to provide if you are to implement a DLR-based language. The idea is to give you a feeling for the amount of work needed to implement a DLR-based language that has the tooling support commonly seen in a language like C#.

Table 1-1. C# versus DLR-based Languages

Facet	C#	DLR-Based Language
Compiler/interpreter	C# compiler	You are free to use your favorite software tools and components to build the language's lexer/parser. You will typically define the language's semantics by mapping the language's syntax tree to DLR expressions.
Runtime	C# uses CLR as the runtime.	Your language can use DLR, CLR as the runtime.*
Code editor	Visual Studio's code editor for C#	The DLR does not provide any feature for implementing a language's source code editor.
Debugger	Visual Studio's debugger for C#	You can leverage the DLR Debugging API.*
Libraries	.NET libraries from Microsoft and third parties are accessible to C# code.	.NET libraries from Microsoft and third parties are accessible to code written in a DLR-based language.*
Unit test framework	NUnit and others	Developers who write code in a DLR-based language can use NUnit and others as the unit test frameworks for testing their code. Most unit test frameworks are software libraries and all .NET libraries are accessible to code written in a DLR-based language.
IDE	Visual Studio	The DLR does not provide any feature for implementing an IDE or integrating into an existing IDE for your language.
Static code analysis	FxCop	The DLR does not provide any feature for implementing a static code analyzer for your language.
REPL, embedded in other languages	C# does not have a REPL console. C# can't be easily embedded in other languages.	DLR Hosting API allows dynamic languages to easily support REPL and to be embedded in other languages.*

In Table 1-1, the cells with an asterisk are the areas in which I think DLR provides the most value. As you can see, a language has a lot of aspects besides syntax and semantics. Syntax and semantics are, of course, the things that define the language. However, to become efficient and productive with a language, you need to be familiar with the various tools and libraries surrounding the language too.

Putting It Together

The previous sections describe what the DLR is by exploring the various concepts, such as runtime, language, and dynamicity. This section will introduce the DLR from a different angle. We will take a high-level look at the major components that make up the DLR. The overview of the major DLR components will serve to orient the rest of the chapters in Part 1 of this book. Figure 1-10 shows the major components that make up the DLR runtime.

Figure 1-10. Major components of the DLR

As the figure shows, the DLR has a compiler and an interpreter. The compiler uses an IL code generator to compile DLR expressions into IL code. The interpreter interprets DLR expressions. Here's a brief description for each of the other components.

Expressions: These are the backbone of the DLR. Almost everything in the DLR centers around DLR expressions. The compiler and interpreter act on DLR expressions. The very important call site caching mechanism and the language interoperability capability that the DLR provides are based on DLR expressions. If you have some experience with LINQ, you will probably be happy to know that these are basically LINQ expressions along with some DLR extensions. It is very likely that the DLR extensions will become a part of LINQ expressions in a later version of .NET. We will explore expressions in great depth in the next chapter.

Call site caching: This is the caching mechanism that makes DLR-based applications run fast. Dynamic languages have been criticized for their performance in comparison to static languages. This is because dynamic languages perform late binding for various actions like method invocation at run time, which is traditionally several orders of magnitude slower than static languages. The call site caching component in DLR solves the problem by caching the results of late binding. The results of late binding are represented as DLR expressions. The caching mechanism is based on an optimization technique called polymorphic inline cache and it's implemented mainly in the CallSiteBinder class. Chapter 3 will dive deep into the details of this DLR component.

Interoperability binders and dynamic objects: This is what enables interoperability between dynamic and static languages. This component uses expressions to represent the results of late binding. It also defines a common type system that consists of twelve late-bound operations to facilitate language interoperability. Binders and dynamic objects are the two kinds of entities in DLR that contain late-binding logic. They interact in a well-established protocol in order to achieve language interoperability. Chapter 4 will cover the interoperability binders. Chapter 5 will look at dynamic objects.

Hosting API: This API allows one language to host (i.e., embed) another language. For example, with this API, we can execute Python code inside a C# application. Once we do that, we generally want to pass some objects from the C# application to the Python code and perhaps receive some objects back from the Python code. That's all possible with the DLR, thanks to the interoperability binders and dynamic objects. We will look at the Hosting API in Chapter 6. In Part 2 of this book, you'll see examples of how to use the Hosting API to allow users to script your applications.

Debugging API: This API is to help you implement a debugger for your DLR-based language. The API is still in its early stage of development and is less mature than the rest of the DLR components. This book will therefore not cover this particular topic. I would refer you to Harry Pierson's weblog at http://devhawk.net for more information on this topic.

Summary

This chapter introduces the DLR by first showing a series of four Hello World examples. It goes on to demonstrate some of the applications you'll develop over the course of this book. After the examples and demonstrations, I describe what the DLR is by explaining some fundamental concepts, such as programming languages, dynamic typing, and late binding, and we looked at the flows of compile time and run time activities. The chapter also includes a high-level discussion on programming language syntax and semantics and a partial survey of tooling support for programming languages. The chapter concludes with a brief description of the key components that make up DLR, with pointers to the chapters that will cover each of these components in more detail. The DLR has wide applicability in many areas of our day-to-day software design and development. This chapter offers a preview, and glimpse of DLR's potential. The rest of the book will dive deep and show the rest of the iceberg underneath.