To use IronPython we need to include all five assemblies that come with it:

The first two assemblies comprise the Dynamic Language Runtime. The next two are specific to IronPython. IronPython.dll implements the Python language syntax and semantics. IronPython.Modules.dll provides a C# implementation of some of the built-in modules that in CPython are written in C.

The last assembly is a bit special; it is there so that .NET projects using IronPython work with both .NET 2.0 and .NET 3.0. The DLR uses extension methods, both because it shares an AST format with LINQ (expression trees) and to provide convenience methods on several parts of the hosting API. The extension methods are decorated with the ExtensionAttribute attribute, which is part of .NET 3.0. All the Microsoft.Scripting.ExtensionAttribute assembly does is provide this attribute so that IronPython can be compiled under .NET 2.0. The important thing is that it must be included in your projects but not directly referenced. If your project is a .NET 3.0 project, then it will use the standard ExtensionAttribute, and if it is a .NET 2.0 project, it will use the one provided by the DLR.

The main entry point for applications that embed IronPython is the Python class in the IronPython.Hosting namespace. It has convenience methods for creating ScriptEngine and ScriptRuntimes preconfigured for working with IronPython.^[3]

Once we have set up a project that references these assemblies, we can extract a Python language engine from their grasp with the following snippet of C#:

using IronPython.Hosting;
ScriptEngine engine = Python.CreateEngine();

The VB.NET equivalent is

Imports IronPython.Hosting
Dim engine As ScriptEngine = Python.CreateEngine()

This code creates a single engine. The Dynamic Language Runtime supports the creation of multiple runtimes within a single application. This can be extremely useful for creating execution contexts that are isolated from each other,^[4] a feature that Resolver One uses to have multiple open but separate spreadsheets.

In the examples that follow we will cover the most common ways of working with the IronPython hosting API, but you often have a choice of several different routes to achieve the same end. For more complete documentation on all the possibilities, you should refer to the source code (this is open source after all) or the “DLR Hosting Spec” document.^[5]

Once we have created an engine, we are ready to execute code, either from a string or directly from a file. At this stage our goal is to create an executable that launches a Python application, so we want to run a file.

To execute code we must introduce two more components from the hosting API, the ScriptSource and the ScriptScope. The ScriptSource can be created from the engine, either from a source file or from code in a string. A ScriptScope represents a Python namespace (a scope) and is also created from the engine.

C# snippets to do this are as follow. First, for executing code from a string:

ScriptSource source;
source = engine.CreateScriptSourceFromString(sourceCode,
   SourceCodeKind.Statements);
ScriptScope scope = engine.CreateScope();
source.Execute(scope);

To execute Python code in a file:

ScriptSource source;
source = engine.CreateScriptSourceFromFile(programPath);
ScriptScope scope = engine.CreateScope();
source.Execute(scope);

Figure 15.1 shows a diagram of the DLR hosting API that we’ve used to get to this point.

Figure 15.1. The major DLR hosting API components used to execute a Python file

This isn’t quite enough, however. In order to properly execute a Python program, we should do a bit more work and set up things like sys.path (the import path) and provide any command-line arguments that were passed to the program.

using IronPython.Runtime;

List argList = new List();
argList.extend(args);
ScriptScope sys = Python.GetSysModule(engine);
sys.SetVariable("argv", argList);

Setting the Search Path

The Python engine has a method called SetSearchPaths, which allows us to pass in an array of paths. These set up sys.path, which is the module search path for import statements and adding references to assemblies.

In order to properly replicate the environment a program runs under when executed with ipy.exe, we need to honor the IRONPYTHONPATH environment variable. This means fetching the environment variable and breaking up a string like C:Python25lib;C:Python25libsite-packages into its component parts by splitting on the semicolons.

Perhaps more important, the directory the Python program is in needs to be in the import path. We can’t assume that this is the current directory, because the application could have been launched from a shortcut or simply from another directory.

Listing 15.2 shows code that creates a List (a generic .NET List, not a Python one) and populates it with the directory containing the Python program plus any paths from IRONPYTHONPATH. It then calls engine.SetSearchPaths with an array from the List.

Example 15.2. Populating the engine import paths from C#

Listing 15.3 does the same thing, but in VB.NET.

Example 15.3. Populating the engine import paths from VB.NET

One thing to be careful of in VB.NET is that the And logical operator does not short circuit (it evaluates both sides of the expression even if the first is False). Before splitting the IRONPYTHONPATH environment variable, we need to check that it is not null and not empty. We need to use the AndAlso operator because we can’t check the length of the string if it is null.

Now we’re ready to execute the code, but with one caveat: because we are executing code in a dynamic language, we need to be able to handle runtime exceptions gracefully.

Handling Python Exceptions

Our .NET code is just a thin wrapper around the Python program, so unhandled exceptions will be fatal anyway, and we could simply let them bubble up. The problem with this approach is that the traceback written to the console will be an incomprehensible CLR traceback, including all the stack frames inside the DLR. What we want is a straightforward Python traceback, and we can get this by catching the exception and using ExceptionOperations to format the traceback.

We have to confess to slightly misleading you earlier; although Execute is a perfectly valid way to run the code in a ScriptSource, it isn’t the best way to run a script that is a full program. Python programs can exit with a return code^[6] (an integer), and if this happens we want to propagate the exit code. Instead of Execute, we can use ExecuteProgram, which returns us the integer exit code (and will create its own scope rather than requiring us to pass one in). Listing 15.4 executes the ScriptSource we created from program.py inside some exception-handling code. If the program terminates normally, it exits with the return code from ExecuteProgram. If an exception is raised, it writes out the formatted exception message and exits with a return code of 1.

Example 15.4. Executing the ScriptSource, handling any exceptions (in C#)

try
{
   ScriptSource source;
   source = engine.CreateScriptSourceFromFile(programPath);
   int result = source.ExecuteProgram();
   return result;
}
catch (Exception e)
{
   ExceptionOperations eo = engine.GetService<ExceptionOperations>();
   Console.Write(eo.FormatException(e));
   return 1;
}

We have pulled all this together into the EmbeddedExecutable example in the downloadable source code.

Something the example does that isn’t shown in the code snippets above is to set debug mode on the engine. This is done by passing in a dictionary of options to the CreateEngine call, with Debug set to true. It causes the IronPython engine to compile Python code in debug mode (without optimizations—so it isn’t recommended for production code). This allows you to use the Visual Studio debugger, including setting breakpoints, on the Python code.

When you run EmbeddedExecutable.exe, it runs program.py, which prints a message along with the command-line arguments it is passed. It then raises an exception to illustrate the exception formatting, as you can see in figure 15.2.

Figure 15.2. An executable application (.exe) that launches a Python program

Embedding IronPython in a custom executable uses only a fraction of the hosting API, but we have used some of the most important classes. The ScriptEngine, ScriptScope, and ScriptSource will accompany us through the rest of the book.

Next we look at some different hosting scenarios and explore more of the API. These build on what we have already learned, but instead of just executing Python code, we actually interact between the host application and the Python environment it is hosting.

The methods that initialize the main execution scope and then execute the Python code are very similar to the code we have already written. There are two important differences, though.

When we were running a Python program, we created a ScriptSource from the source code file and called ExecuteProgram to run it. ExecuteProgram executes code in its own execution scope. However, if we execute code in an explicitly created ScriptScope, we can then access variables, classes, and the like that have been created by the executed code. Conversely, if we want to provide objects for the code to use, we can place them in the scope prior to execution.

The second difference in the embedding code we are about to write is the way that code is executed. In the snippets we have just seen, a ScriptSource is created from the source code and then executed directly with the scope. If we are going to execute the code several times, which is entirely likely in a hosting situation, then we can optimize by compiling the code from the ScriptSource.

Calling script.Compile() returns us a CompiledCode object that we can use in place of the ScriptSource.

ScriptScope and CompiledCode are two highly reusable components. You can create multiple scopes and execute the same code in the different scopes or create a single scope and execute different code with access to the same objects.

The ScriptScope is a DLR class and is not tied to IronPython. You can execute Python code to create an object graph in a scope and then execute Ruby code in the same scope and with access to the same objects.^[8] This permits some very interesting interoperability stories. Where IronRuby (or Managed JScript or IronScheme or ...) uses IronPython objects, they retain their behavior as Python objects but are still usable from these other languages. There are some restrictions in the ways they can interoperate; it is unlikely that Python classes will ever be able to inherit from Ruby classes or vice versa, for example.^[9] It should still be possible for dynamic languages running on .NET to share libraries, though. Python on Rails or Ruby on Django, anyone?

From a hosting point of view, the interesting thing we can do with scopes is to set variables in and fetch variables out of them. This means that you can publish an object model from the hosting application into the scope for user code to work with (call methods, add handlers to events, and so on). After executing Python code you can then fetch objects it has created out of the scope.

The methods for working with names contained in a scope are shown in table 15.2.

You also have read-only access to all the variable names as IEnumerable of string through the public VariableNames property, and the variables themselves as IEnumerable of KeyValuePairs (containing string, object pairs) through the Items property.

The most important of these methods are the three that allow you to set and fetch variables in the scope:

Setting variables is straightforward. You call SetVariable with any string and any object. If you want Python code to actually have access to the variables you set, then you should restrict the strings (variable names) to be valid Python identifiers, but the scope doesn’t enforce this.^[10]

Fetching variables has more complications associated with it, mainly because of the impedance mismatch of interacting with a dynamic language from a statically typed language. So long as the variable exists—and after executing arbitrary Python code there’s no guarantee of that—C# and VB.NET insist on knowing the type before they will allow you to do anything useful with it.

For known types you have a choice of checking that the variable exists with ContainsName and use the generic version of GetVariable to fetch it from the scope. To fetch a string you use GetVariable<string>(name) from C# or GetVariable(OfString)(name) from VB.NET. Alternatively you can use TryGetVariable, which takes an out parameter that will be null (nothing) after the call if the variable doesn’t exist.^[11] From C# you will then need to cast the value to the known type after fetching it. TryGetVariable returns a Boolean indicating success or failure of attempting to fetch the variable.

If you are executing arbitrary code you will, therefore, need code that can handle the variable not existing or being the wrong type. If you are executing known code rather than arbitrary code, then it is fine to do any necessary error handling within the Python code and be able to guarantee that the variable exists and is of the expected type.

For .NET types, once you’ve pulled them out of the scope you have full access to them as if they were created from C#/VB.NET. If they are dynamic objects, such as functions or classes, you can still use them—but you have to use mechanisms provided by the Dynamic Language Runtime to perform operations on them.^[12] This is something that we will look at later in the chapter.

In the meantime we now have all the pieces we need to set variables in a scope, execute code in that scope, and then fetch objects back out. Listing 15.5 is the start of an Engine class in C#. You instantiate it with Python source code as a string.

Example 15.5. Creating an execution scope with access to contained variables (in C#)

The Execute method catches any exceptions raised in the Python code (writing the Python exception to standard out) and returns a Boolean indicating success or failure. The SetVariable and TryGetVariable methods on the engine merely delegate to methods on the ScriptScope. Handling error conditions, such as the variable not existing or being of the wrong type, is up to the caller.

This code does something that our last example didn’t. Now that we know how to set variables in our execution scope, we can set the __name__. Python code would expect __name__ to exist, and for the top-level script it would expect it to be set to __main__. The reason why we didn’t do this when creating the executable is that engine.CreateScriptSourceFromFile implicitly sets the name in the scope to the name of the file (minus the .py).^[13] The advantage of using CreateScriptSourceFromFile is that it doesn’t read the whole file at once. If your top-level program depends on the name being set to __main__, then you can have the best of both worlds by using the three-argument form: engine.CreateScriptSourceFromFile(path, Encoding.Default, SourceCodeKind.Statements).

This allows you to execute the ScriptSource in a scope with an explicit __name__ set, without it being implicitly overridden because it is from a file.

With what you know already you could probably achieve almost everything you need when embedding IronPython. Magically injecting variables is not (always) the most elegant way of exposing objects to your hosted code. A much nicer solution is to make your objects available either in Python modules or in assemblies that the Python code can import in the usual way.

Adding references to assemblies is done with the LoadAssembly method on the ScriptRuntime. This takes an actual assembly object, so we need to use the System.Reflection API to obtain it.

We can use this to overcome another minor limitation when embedding IronPython. ipy.exe adds references to mscorlib.dll and System.dll (the core assemblies containing the System namespace) for us. This means that import System, or variants, can be executed without explicitly having to add references to these assemblies. Code running inside the default embedded IronPython engine will need to manually add the references before being able to import from System. We can solve this by getting Assembly objects from types in each of the two assemblies. We can do this in C# with this:^[14]

_runtime.LoadAssembly(typeof(String).Assembly);
_runtime.LoadAssembly(typeof(Uri).Assembly);

Which translates to this in VB.NET:

_runtime.LoadAssembly(GetType(String).Assembly)
_runtime.LoadAssembly(GetType(Uri).Assembly)

The BasicEmbedding example also includes a ClassLibrary.dll assembly containing a class with a couple of static methods (shared functions in VB.NET–speak) that write to stdout. We can add a reference to this assembly by first loading it with Assembly.LoadFile from the same directory as the executable. This time we have the example code in VB.NET, shown in listing 15.6.

Example 15.6. Adding a reference to an assembly on the ScriptRuntime (in VB.NET)

Once we’ve added a reference to the assembly, Python code running in the hosted engine is free to import from its namespace(s).

If the object model you want to expose isn’t contained conveniently in a single assembly, or you want to construct it from live objects at runtime, then an alternative is to construct a Python module and add that to the runtime instead.

In Python, import statements first check to see if the requested module is already in sys.modules. The embedded equivalent is runtime.Globals, and just like sys.modules it doesn’t have to be a “real” Python module you put in there but any object you want Python code to be able to import.

ScriptRuntime.Globals is actually our old friend the ScriptScope. This means that you set objects in it using the same SetVariable method we have already used. You can put any object into there, and import statements executed in the embedded engine will fetch them. Using this technique is another way to make a host object model available to user code.

The challenge is that we do want to create a real Python module to put in there. We could just create and populate a new ScriptScope and publish that. Under the hood IronPython modules use a ScriptScope to store the namespace, and it would behave like a module when imported. However, ScriptScope doesn’t have the right repr, and it has a few other minor differences, so it will seem a bit odd if the user does introspection on a ScriptScope published as a module. The answer is to use the Scope object, which as far as IronPython is concerned is a real Python module. We construct a Scope object from a ScriptScope using HostingHelpers.GetScope.^[15]

The following snippet of VB.NET creates a ScriptScope called inner in which we set a string with the name HelloWorld. This is wrapped in a Scope object that is then published into the runtime globals.

Dim _module As Scope
Dim inner As ScriptScope

inner = _engine.CreateScope()
inner.SetVariable("HelloWorld", "Some string...")

_module = HostingHelpers.GetScope(inner)
_runtime.Globals.SetVariable("Example", _module)

Code running in the embedded engine can either execute import Example and access HelloWorld as a module attribute or execute from Example import HelloWorld to get direct access to the string we set in inner.

We’ve now covered all the major classes necessary for a wide range of different embedding scenarios. Figure 15.3 summarizes what we learned so far. It shows the core classes that we have worked with and the relationship between them and their useful members. Our core Engine class is now basically complete, with only one minor modification needed. Since we know how to make modules available for importing, we turn the scope in which we execute the main script into a proper module.

Figure 15.3. Core hosting classes^[16]

This C# snippet does this and puts the module into the runtime globals with the name __main__:

_scope = _engine.CreateScope();
_scope.SetVariable("__name__", "__main__");

Scope _main = HostingHelpers.GetScope(_scope);
_runtime.Globals.SetVariable("__main__", _main);

The advantage of doing this is that the Python code import __main__ now does the right thing. This is not a common thing to do but is used, for example, by unittest.main() to introspect the main execution scope and find all TestCase classes.

The example project uses the Engine class by embedding a Python script as a resource. Let’s see how it uses the engine.

This example executes a Python file that is stored as an embedded resource compiled into the main assembly. This can be a useful way of preventing Python code your application depends on from being modified in your distributed application. Even though it isn’t distributed as plain text, it isn’t an effective way of keeping the source code secret, as it will be easily discoverable within the assembly.^[17]

Adding a Python source file as an embedded resource to a Visual Studio project is as simple as adding the file (either from an existing file or adding a new text file and renaming) and setting the Build Action to Embedded Resource, as shown in figure 15.4.

Figure 15.4. Creating embedded resources in Visual Studio

Listing 15.7 shows the C# code to retrieve the source code from the embedded resource as a string.

static string GetSourceCode()
{
   Assembly assembly = Assembly.GetExecutingAssembly();
   string name = "BasicEmbedding.source_code.py";
   Stream stream = assembly.GetManifestResourceStream(name);
   StreamReader textStreamReader = new StreamReader(stream);
   return textStreamReader.ReadToEnd();
}

The important call here is assembly.GetManifestResourceStream(name), which gives us access to our embedded source file as a stream.

Our example retrieves the source code and uses the Engine we have created to execute it. Before executing it sets a variable (imaginatively called variable) into the execution scope and fetches it out again after execution. Listing 15.8 shows the full code in VB.NET.

Example 15.8. Using the Engine to execute Python code, with error handling (in VB.NET)

engine.Execute does the error handling for us, returning a Boolean indicating whether or not the execution succeeded. Assuming we can’t trust our code not to fail, we need to check this result and handle the failure. Because we are actually executing code from an embedded resource and not arbitrary user code, we could skip the error checks if they aren’t necessary. If the code executes correctly, it fetches the variable variable back out of the execution scope (handling the case when it doesn’t exist in the scope) and writes it out to the console. Because we are writing it out to the console, we fetch it as an object, neatly bypassing the problem of having to know what type it is.

Unfortunately, this type problem we skipped over is quite a big issue for using embedded IronPython for anything practical. Although we already have most of the pieces of the jigsaw, the next section puts them together (and shows one way of resolving the type problem) by implementing a plugin system for a .NET application.

In order to solve the type problem, we can provide a base class that plugin classes must inherit from. On the .NET side we can handle user plugins as this base class, and the compiler is happy—knowing what methods and properties are available.^[18]

We also have to provide a mechanism for the plugins to be added to the application. We can do this with a PluginStore class, also accessible to user code, which acts as a registry for plugins.

Our PluginBase class is instantiated with a name, which will be used for the toolbar button. It also provides an Execute method, which does nothing on the base class but will be called with the textbox in real plugins (when the corresponding toolbar button is clicked).

On the .NET side, where we need to interact with the user plugins, we can use the PluginBase type. Listing 15.9 shows an implementation in C#.

Example 15.9. A base class for user plugins to inherit from (in C#)

The PluginStore class is equally simple. It has a public static (shared) method, AddPlugin, which takes a PluginBase instance. This adds the plugin to a list accessed as the Plugins property. This list is marked as internal (friend), which means that only classes in the same assembly can access the list and it isn’t exposed to user code. Listing 15.10 shows the PluginStore class in VB.NET.

Public Class PluginStore

   Private Shared _plugins As List(Of PluginBase) _
      = New List(Of PluginBase)()

   Friend Shared ReadOnly Property Plugins() _
      As List(Of PluginBase)
      Get
         Return _plugins
      End Get
   End Property

   Public Shared Sub AddPlugin(ByVal plugin As PluginBase)
      _plugins.Add(plugin)
   End Sub
End Class

In fact, the PluginStore is the only public class in the main assembly. We can add the main assembly to the runtime (along with the assembly containing the PluginBase), and user code can call PluginStore.AddPlugin without being able to access any of the application internals that we haven’t explicitly exposed.

Listing 15.11 shows the Python code that creates a new plugin and adds it to the PluginStore.

from Plugins import PluginBase
from EmbeddingPlugin import PluginStore

class Plugin(PluginBase):
  def Execute(self, textbox):
    textbox.Text += "Plugin 1 called
"

plugin = Plugin("Plugin 1")
PluginStore.AddPlugin(plugin)

Our application can execute all the user scripts and then access any successfully added plugins via PluginStore.Plugins. Let’s look at how our application loads the user scripts.

Our application uses a very simple mechanism to load user scripts. Alongside the executable is a directory called plugins. When the application starts, it executes all the Python files in this directory.

As we are executing user code we obviously need to be tolerant of errors, and to make things more interesting, let’s see how we can handle specific Python errors differently.

The SyntaxErrorException is raised when we attempt to execute invalid code. It lives in the Microsoft.Scripting namespace. It is an exception shared by all DLR languages. The Python-specific exceptions live in the IronPython.Runtime.Exceptions namespace, so with the right using or Imports directive we can catch specific Python errors. Listing 15.12 shows the code that loads and executes all the plugins. Any errors in executing plugin code are caught, and a message box is displayed to the user, but syntax errors and SystemExit exceptions^[19] are treated differently with a custom message. Figure 15.6 shows the error message shown to users for syntax errors.

Figure 15.6. Custom error message for syntax errors in plugin code

Listing 15.12 is the C# code that loads and executes the scripts from the plugins directory.

Example 15.12. Finding and executing user plugins (in C#)

The ShowError method may seem slightly odd: the caller has to extract the filename from the full path instead of it being done inside the method. It is written this way because we reuse ShowError to show a different kind of message when we call the plugins.

Before we finish off the application by hooking up the plugins to the toolbar buttons, we look at how we can improve the development process for the user writing the plugins.

This example is a Windows Forms application, which makes for much better screen-shots. It does have one disadvantage, though; it means that print statements inside the user code are lost because there is no standard output for them to go to. This could make debugging plugins much harder for our users. Luckily, DLR runtimes support diverting the standard output and error streams, and in this example we send them to the application textbox.

The API for diverting the output and error streams^[20] is through runtime.IO.SetOutput and runtime.IO.SetError methods that take a .NET Stream and an encoding. To use these methods, we need a Stream that diverts everything written to it back to the textbox.

Stream is an abstract class—inheriting from it is easy but requires implementing a tedious number of methods and properties. For this example we’ve chosen to inherit from MemoryStream and override the Write method. This is an abuse of MemoryStream, but it works fine.

Listing 15.13 shows the PythonStream class and setting an instance onto the runtime to divert both standard output and standard error.

internal class PythonStream: MemoryStream
{
   TextBox _output;
   public PythonStream(TextBox textbox)
   {
      _output = textbox;
   }

   public override void Write(byte[] buffer, int offset,
                      int count)
   {
      string text = Encoding.UTF8.GetString(buffer, offset,
         count);
      _output.AppendText(text);
   }
}

public void SetStreams()
{
   PythonStream stream = new PythonStream(_box);
   _runtime.IO.SetOutput(stream, Encoding.UTF8);
   _runtime.IO.SetErrorOutput(stream, Encoding.UTF8);
}

Minor points are worth noticing from this listing. When we set the streams on the runtime we need to specify an encoding; so in our custom stream we need to use the same encoding to decode the bytes back into a string. The stream is constructed with a textbox, and the Write method simply calls AppendText to add new text.

The “Loading Plugin 1” and other messages shown in figure 15.5 actually come from print statements in the plugin code. When the plugin is loaded (executed), anything written to standard out appears in the main textbox.

The final thing we need to do is to create a toolbar button per plugin and hook up its Click event to call the appropriate Execute method.

Once the plugins have all been loaded, we can iterate over PluginStore.Plugins and add a button and handler for each one. The code to do this from C# is shown in listing 15.14.

int index = 0;
foreach (PluginBase plugin in PluginStore.Plugins)
{
   ToolStripButton button = new ToolStripButton();
   button.ToolTipText = plugin.Name;
   button.Text = plugin.Name;

   int pluginIndex = index;
   button.Click += delegate {
      ExecutePluginAtIndex(pluginIndex);
   };

   pluginToolStrip.Items.Add(button);
   index++;
}

public void ExecutePluginAtIndex(int index)
{
   PluginBase plugin = PluginStore.Plugins[index];

   try
   {
      plugin.Execute(_textbox);
   }
   catch (Exception e)
   {
      string msg = "Error executing plugin "{0}"";
      ShowError(msg, plugin.Name, e);
   }
}

In C# we can use a closure to call the right plugin when its handler is invoked. By turning the index into a variable local to the scope of the foreach loop (the pluginIndex variable), the anonymous delegate will invoke ExecutePluginAtIndex with the correct index when its button is clicked.

The VB.NET code, shown in listing 15.15, has to be slightly different.

Dim index As Integer = 0
For Each plugin As PluginBase In PluginStore.Plugins

   Dim button As ToolStripButton = New ToolStripButton()
   button.ToolTipText = plugin.Name
   button.Text = plugin.Name

   Dim handler As ButtonHandler
   handler = New ButtonHandler(engine, index, button)
   pluginToolStrip.Items.Add(button)
   index += 1
Next

Friend Class ButtonHandler
   Dim _engine As Engine
   Dim _index As Integer

   Public Sub New(ByVal engine As Engine, _
              ByVal index As Integer, _
              ByVal button As ToolStripButton)
      _engine = engine
      _index = index

      AddHandler button.Click, AddressOf ClickHandler
   End Sub

   Public Sub ClickHandler(ByVal sender As Object, _
                    ByVal e As EventArgs)
      _engine.ExecutePluginAtIndex(_index)
   End Sub
End Class

Public Sub ExecutePluginAtIndex(ByVal index As Integer)
   Dim plugin As PluginBase = PluginStore.Plugins(index)

   Try
      plugin.Execute(_box)
   Catch e As Exception
      Dim msg As String = "Error executing plugin ""{0}"""
      ShowError(msg, plugin.Name, e)
   End Try
End Sub

Because we don’t have the luxury of closures in VB.NET, we have to use a class to capture the state (which is what the compiler does under the hood for us in C# anyway). For every plugin a new ButtonHandler instance is created, which has a ClickHandler to be called when the button is used.

In the previous two sections of this chapter we have gone through two different examples of embedding IronPython. The first was a gentle amble through the hosting API that introduced us to all the major classes. The example that we have just completed showed us how to use embedded IronPython for a practical purpose, in the form of application plugins. This example solved the mismatch between dynamic and statically typed languages by using a specific type for plugins. We can also use more general techniques to interact with dynamic objects and, in the process, extend IronPython to some new use cases.

The simplest example of this is to use IronPython to evaluate individual expressions. IronPython makes a great calculator!^[21]

So far, whenever we have executed Python code from a string we have passed in the enumeration member SourceCodeKind.Statements. This member has a sister, SourceCodeKind.Expression, that allows us to evaluate an expression and return an object. It is used in this snippet of C# to evaluate a simple mathematical expression:

string code = "2 + 3 + 5";
ScriptSource source;
source = runtime.CreateScriptSourceFromString(code,
   SourceCodeKind.Expression);
int result = source.Execute<int>(scope);

This code uses the generic form of Execute that allows us to specify the type of the returned object. Using the generic version requires us to pass in an explicit scope to Execute. If we wanted to use the default scope, we could instead cast the returned object to retrieve the integer.

Since we are retrieving results with specific types, we can use the System.Func delegate to define functions in Python and call them from other .NET languages. This delegate is a standard part of .NET 3.5 (C# 3.0), but it is also provided by IronPython 2, so you can happily use it in .NET 2.0 projects.

The simplest example of this is to evaluate an expression that returns us a lambda function, as shown in listing 15.16.

string code = "lambda x, y: x * y";

ScriptSource source;
source = engine.CreateScriptSourceFromString(code,
   SourceCodeKind.Expression);

Func<int, int, int> lambda;
lambda = source.Execute<Func<int, int, int>>(scope);

int result = lambda(9, 8);
Console.WriteLine("9 * 8 = {0}", result);

Listing 15.17 shows the equivalent VB.NET code.

Dim code As String = "lambda x, y: x * y"

Dim source As ScriptSource
Dim lambda As Func(Of Integer, Integer, Integer)
source = engine.CreateScriptSourceFromString(code, SourceCodeKind.Expression)

lambda = source.Execute( _
Of Func(Of Integer, Integer, Integer))(scope)

Dim result As Integer = lambda(9, 8)

Console.WriteLine("9 * 8 = {0}", result)

Func is generic, with type specifiers for the arguments and the return value. This example specifies a function that takes two integers as arguments and returns an integer: Func<int, int, int>. The last type specified is always the return type. There are convenient definitions for functions that take up to eight arguments (and if that isn’t enough, you need to rethink your function API!).

Instead of evaluating an expression to create a lambda, we could execute a function definition in a scope and pull the function out by name. This would work with any callable object, and we could pull them out as any form of delegate—allowing the ScriptScope to do the language conversion when we pull out the value.

As well as allowing you to use dynamically created functions, you could use Python functions to provide a .NET API to Python libraries.

We can use functions because they have a corresponding type defined in C#, and the same is true of the other Python types. The built-in Python types live in the IronPython.Runtime namespace. The very basic types, like strings and integers, aren’t there because IronPython uses the standard .NET types for these. Types that are available include the following:

Most of the normal Python methods are available on these types, including the magic double-underscore methods. You can explore the types and their members using Visual Studio 2008 Object Browser or Reflector.^[23] Figure 15.7 shows some of the members on the Python List type.

Figure 15.7. The Python List type viewed from the Visual Studio 2008 Object Browser

We can retrieve these objects by type from a Python scope or create and use them directly. When we created a custom executable for a Python program, we created a Python List and set it on the language context as sys.argv. Listing 15.18 shows creating a tuple in Python and then using it from C#.

string code = "('hello', 'world')";
ScriptSource source;
source = engine.CreateScriptSourceFromString(code,
   SourceCodeKind.Expression);

PythonTuple tuple = source.Execute<PythonTuple>(scope);
Console.WriteLine("tuple = ({0}, {1})", tuple[0], tuple[1]);

These techniques are great for the built-in Python types, and for objects with known types, but they don’t allow us to use classes defined in Python. Fortunately, there are other techniques we can use with Python classes and instances.

IronPython (and IronRuby) classes are not .NET classes. IronPython classes are .NET objects—instances of IronPython.Runtime.Types.PythonType, the Python type metaclass.^[24] This is because Python classes can have members added and removed at runtime. You can even change their base classes dynamically, things that we couldn’t do with .NET classes.

Python classes are still usable from C# and VB.NET; we can actually solve the type problem by ignoring it! We can keep objects as objects on the .NET side and use the Python engine to perform operations like creating instances and calling methods.

The mechanism for doing this is to use ObjectOperations, a class that provides dynamic operations for DLR objects. The ScriptEngine exposes this as the Operations property, returning an ObjectOperations instance bound to the semantics of the engine’s language.

ObjectOperations is a class with many useful methods, and again the best way to find out what it can do is to explore it with Reflector or the Visual Studio Object Browser. It knows how to perform comparisons and mathematical operations on dynamic objects, following the semantics of the language for those operations. In the case of Python that means autopromoting integers to longs if necessary and using the __add__ method for addition where appropriate, and so on.

More important, ObjectOperations knows how to call objects and set and fetch members. Using these capabilities alone we can achieve most of what we might want to do with dynamic objects. Listing 15.19 shows how to fetch a Python class out of an execution scope, create an instance of that class, and call a method on it.

Example 15.19. Creating class instances and calling methods using ObjectOperations

Calling the method is a little cumbersome, but if we were working with known Python classes, it would be easy to abstract away with a thin wrapper layer.

We create an instance of a class in the same way we do in Python, by calling the class. This is done with ops.Call, and afterward we fetch a method from the instance with ops.GetMember. GetMember takes the instance and the name of the member we want to fetch as a string. This mirrors the way we use the getattr function from Python.

Having fetched the method we call it, again with ops.Call. Any arguments passed to ops.Call are passed in as arguments to the object we are calling; the overload we are using is Call(object, params object[]). We cast the objects to known types only at the point at which we actually need them, in this case the return value from calling the method. GetMember also has a generic version, which we can call if we care about the type of object it will return.

The corresponding partner to GetMember is SetMember. Its signature matches setattr; ops.SetMember(object, name, value) where name is a string specifying the member to set.

Table 15.3 lists methods of ObjectOperations, including the three we have already discussed.

It may not be immediately obvious why we might need ConvertTo and TryConvertTo when we can always cast objects. We can use them to take advantage of some of the magic that IronPython does on our behalf, particularly when using Python functions as event handlers. Under the hood IronPython wraps the function as a delegate for us, and we can use ObjectOperations.ConvertTo to do the same from the .NET side. The following snippet shows how to do this from C#:

ObjectOperations ops = engine.Operations;
object function = scope.GetVariable("function");
SomeObj.SomeEvent += ops.ConvertTo<DelegateType>(function);

Remember that if you want to be able to unhook the handler, you’ll need to keep a reference to the delegate it returns!

As well as being able to fetch and call methods on dynamic objects, we can use the Python built-in functions on them.

If we have a collection of objects created from Python code and we want to add them up, we have several choices of how to do it. We could set them in an execution scope with names and evaluate an expression that adds them up, or we could use ObjectOperations.Add in a loop. We have a third choice as well; Python has a perfectly good built-in function for adding things up: sum.

Python has a wide range of useful built-in functions, and in IronPython they are implemented as static methods on IronPython.Runtime.Builtin. Some of the builtin functions require a CodeContext; we’ll see how to provide this when we look at working with built-in modules. sum is perhaps not the most useful example to pick, but some of the built-in functions are very useful for working with dynamic objects.

For example, if we have obtained (by whatever means) a dynamic object that we believe to be an instance of a particular class, we can confirm this by using the isinstance function. Listing 15.20 shows an example of using both the isinstance and the issubclass functions from C#.

Example 15.20. Using the built-in functions isinstance and issubclass from C#

This example creates two Python classes and fetches them both out of the scope. It creates an instance of the first one (Something) and then performs various checks on the instance and the classes using Builtin.isinstance and Builtin.issubclass.

One difference about the way we use Python classes in this example is that we are specifically pulling them out as PythonType rather than object. isinstance takes two objects, but issubclass requires the first argument to be a PythonType (new style class) or OldClass (guess).

We’ve now used the built-in types and the built-in functions, but there is one more class of Python built-in that we haven’t used directly with embedded IronPython. From the title of this section it won’t come as a shock to you to hear that these are the built-in Python modules.

The built-in modules are implemented in the IronPython.Modules assembly. In there you will find a single entry for each module (ClrModule, PythonDateTime, PythonSocket, and so on) plus one for each type that the module exports. Functions in the module are defined as static methods on the module object.

Listing 15.21 uses the Python pickle module^[25] to serialize and deserialize a Python dictionary. It then checks that the serialization and deserialization have worked, using ObjectOperations.Equal to compare the original dictionary and the deserialized object. In order to use the pickle loads and dumps functions, we must first create a CodeContext.^[26]

Example 15.21. Serializing and deserializing Python objects from C# with pickle

The call to PythonPickle.dumps takes a CodeContext, a Python object, a protocol, and a bin argument. First we construct a CodeContext with a new Scope and the appropriate LanguageContext. Creating CodeContexts like this is useful for working with several of the built-in functions and modules.

This example uses protocol 0, which ensures that the resulting pickle is serialized as ASCII rather than using a binary format. Storing binary data in Unicode .NET strings is a recipe for pain, and it means we can pass in null for the bin argument.

dumps returns an object, which we can cast to a string. If we feed this string back into PythonPickle.loads (along with the context), it deserializes the string back into a Python dictionary for us. This could be useful for persisting state from the dynamic part of an application.

The DLR hosting API is larger than we could hope to cover in a single chapter. Topics we haven’t had a chance to look at include the Dynamic Language Runtime support for creating engines and executing code inside AppDomains. This is a good way of limiting what user code is able to do when run inside your application. It also has a Platform Adaptation Layer (PAL), used by Silverlight, that allows you to customize the way files are opened and imports are resolved and even to execute code in custom namespaces that can provide or change names on demand.

Not only that, but there is an enormous number of ways you could use embedded IronPython. What we have managed to do, though, is look at some of the most common ways of using IronPython, and most of the numerous alternatives will bear some similarity to the scenarios we have explored. What I really hope you take away from this chapter is a kick-start in being able to imagine what is possible and knowing how to start exploring. Before we finish, there is some big news about how you will interact with dynamic languages in future versions of the .NET framework.

At the 2008 Microsoft PDC conference, Jim Hugunin and Anders Hejlsberg announced^[27] that the DLR would be integrated into version 4.0 of the .NET framework. Alongside this there will be changes to C# and Visual Basic to introduce dynamic features that use the DLR.

The major change in C# 4.0, at least the one that is relevant to us, is the addition of the dynamic keyword. This is a static declaration to the compiler that operations on the object are to be handled dynamically at runtime! Operations on objects declared as dynamic will be delegated to the DLR. For ordinary .NET objects the DLR uses reflection (just as it does inside IronPython), but it also enables some things not normally possible from C#. These include duck typing, the use of late-bound COM, and the creation of fluent APIs,^[28] like XML or DOM traversal using element names as object attributes.

More important, it allows you to receive objects from a DLR language engine and use them as dynamic objects. Objects created by IronPython or IronRuby can be used from C# while retaining their behavior as Python and Ruby objects.

The C# Future documentation^[29] gives this example of using the new dynamic keyword. All of the uses of d shown here will be done by the DLR:

dynamic d = GetDynamicObject(...);
d.M(7); // calling methods
d.f = d.P; // getting and settings fields and properties
d["one"] = d["two"]; // getting and setting through indexers
int i = d + 3; // calling operators
string s = d(5,7); // invoking as a delegate
int a = d; // assignment conversion

The final operation creates a typed object (a) from the dynamic object d. As with the examples you’ve been working on in this chapter, these operations could raise runtime errors, and so the sort of error handling that we’ve been discussing will still be needed.

The bottom line is that the hosting APIs make it easy to work with dynamic languages, but interacting with dynamic objects will get a whole lot easier.