|
Embedding the IronPython engine is one of the major reasons for wanting to use IronPython. It provides a ready-made scripting language engine for embedding into applications. You can use it to provide user scripting and plugins or integrate Python code into systems written in other languages. You could even use it to provide user scripting for Silverlight applications.
Embedding the IronPython engine means turning the world, as we have seen it so far, inside out. Instead of using .NET objects from inside IronPython, we host the language engine inside C# or VB.NET—and interact with Python objects from these languages. This means that we make objects available for the Python code to work with and then work with objects created from Python back on “the other side.”
Although we work specifically with IronPython, much of what we look at is relevant to the other DLR languages as well.
There are many different ways of embedding IronPython, corresponding to the myriad of use cases to which you might apply it. In this chapter we take a leisurely stroll through some of the more common hosting scenarios. Not only will this cover many of the straightforward things you might want to achieve, but it will equip you to experiment with more esoteric uses to which you might bend IronPython.
IronPython is an enormously powerful tool for use in .NET applications. As well as providing ways to extend and script applications, it also allows you to create custom DSLs[1] where business rules can be stored as text without the application having to be recompiled. We’ll be covering all of these use cases but starting with one of the simpler use cases: creating an executable that launches a Python application.
One of the most basic ways of embedding IronPython is to create an executable file that executes a Python file. This allows you to distribute a Python application, where the main entry point is a standard .exe file. Because this uses only a subset of the API, it makes a great first example. Before we tackle the whole code for creating a custom executable, let’s look at the major components in the hosting API, starting with the IronPython engine itself.
To use IronPython we need to include all five assemblies that come with it:
Microsoft.Scripting.dll
Microsoft.Scripting.Core.dll
IronPython.dll
IronPython.Modules.dll
Microsoft.Scripting.ExtensionAttribute.dll
The first two assemblies comprise the Dynamic Language Runtime. The next two are specific to IronPython. IronPython.dll implements the Python language syntax and semantics. IronPython.Modules.dll provides a C# implementation of some of the built-in modules that in CPython are written in C.
The last assembly is a bit special; it is there so that .NET projects using IronPython work with both .NET 2.0 and .NET 3.0. The DLR uses extension methods, both because it shares an AST format with LINQ (expression trees) and to provide convenience methods on several parts of the hosting API. The extension methods are decorated with the ExtensionAttribute
attribute, which is part of .NET 3.0. All the Microsoft.Scripting.ExtensionAttribute
assembly does is provide this attribute so that IronPython can be compiled under .NET 2.0. The important thing is that it must be included in your projects but not directly referenced. If your project is a .NET 3.0 project, then it will use the standard ExtensionAttribute
, and if it is a .NET 2.0 project, it will use the one provided by the DLR.
The main entry point for applications that embed IronPython is the Python
class in the IronPython.Hosting
namespace. It has convenience methods for creating ScriptEngine
and ScriptRuntime
s preconfigured for working with IronPython.[3]
Once we have set up a project that references these assemblies, we can extract a Python language engine from their grasp with the following snippet of C#:
using IronPython.Hosting; ScriptEngine engine = Python.CreateEngine();
The VB.NET equivalent is
Imports IronPython.Hosting Dim engine As ScriptEngine = Python.CreateEngine()
This code creates a single engine. The Dynamic Language Runtime supports the creation of multiple runtimes within a single application. This can be extremely useful for creating execution contexts that are isolated from each other,[4] a feature that Resolver One uses to have multiple open but separate spreadsheets.
In the examples that follow we will cover the most common ways of working with the IronPython hosting API, but you often have a choice of several different routes to achieve the same end. For more complete documentation on all the possibilities, you should refer to the source code (this is open source after all) or the “DLR Hosting Spec” document.[5]
Once we have created an engine, we are ready to execute code, either from a string or directly from a file. At this stage our goal is to create an executable that launches a Python application, so we want to run a file.
To execute code we must introduce two more components from the hosting API, the ScriptSource
and the ScriptScope
. The ScriptSource
can be created from the engine, either from a source file or from code in a string. A ScriptScope
represents a Python namespace (a scope) and is also created from the engine.
C# snippets to do this are as follow. First, for executing code from a string:
ScriptSource source; source = engine.CreateScriptSourceFromString(sourceCode, SourceCodeKind.Statements); ScriptScope scope = engine.CreateScope(); source.Execute(scope);
To execute Python code in a file:
ScriptSource source; source = engine.CreateScriptSourceFromFile(programPath); ScriptScope scope = engine.CreateScope(); source.Execute(scope);
Figure 15.1 shows a diagram of the DLR hosting API that we’ve used to get to this point.
This isn’t quite enough, however. In order to properly execute a Python program, we should do a bit more work and set up things like sys.path
(the import path) and provide any command-line arguments that were passed to the program.
Command-line arguments in a .NET application are passed into the Main
method of our application as a string array. Python programs expect sys.argv
to be a list of strings, where the first entry is the program filename and the entries following are the command-line arguments.
The Python
class provides easy access to the sys
module for a Python engine with the GetSysModule
method. This method takes the engine as an argument and returns a ScriptScope
. Listing 15.1 extracts the sys
module and sets the command-line arguments.
Example 15.1. Setting the command-line arguments for Python code from C#
using IronPython.Runtime; List argList = new List(); argList.extend(args); ScriptScope sys = Python.GetSysModule(engine); sys.SetVariable("argv", argList);
The value of sys.argv
needs to be a Python list. Naturally the Python list type in IronPython is implemented in C#, so we can use it from our embedding code—including using Python methods like append
and extend
to add elements.
We’ve now used two convenience methods from the Python
class. It has many more, the most useful of which are listed in table 15.1.
Table 15.1. Methods on the Python
class
Method name | Purpose |
---|---|
| Creates a new |
| Creates a new |
| Given a |
| Gets a |
| Gets a |
| Gets a |
| Imports the Python module by the given name and returns its |
Both CreateRuntime
and CreateEngine
optionally take a dictionary of options to configure the engine/runtime. GetSysModule
, GetBuiltinModule
, GetClrModule
, and ImportModule
can all be called with either a ScriptRuntime
or a ScriptEngine
. These four methods are also extension methods, so if you’re using .NET 3.5 you’ll be able to do someEngine.GetSysModule()
after using or importing IronPython.Hosting
.
Back to our example: as well as passing in the command-line arguments, we also need to make sure that the import path is set up correctly for the Python program.
The Python engine has a method called SetSearchPaths
, which allows us to pass in an array of paths. These set up sys.path
, which is the module search path for import statements and adding references to assemblies.
In order to properly replicate the environment a program runs under when executed with ipy.exe, we need to honor the IRONPYTHONPATH
environment variable. This means fetching the environment variable and breaking up a string like C:Python25lib;C:Python25libsite-packages
into its component parts by splitting on the semicolons.
Perhaps more important, the directory the Python program is in needs to be in the import path. We can’t assume that this is the current directory, because the application could have been launched from a shortcut or simply from another directory.
Listing 15.2 shows code that creates a List (a generic .NET List, not a Python one) and populates it with the directory containing the Python program plus any paths from IRONPYTHONPATH
. It then calls engine.SetSearchPaths
with an array from the List.
Listing 15.3 does the same thing, but in VB.NET.
One thing to be careful of in VB.NET is that the And
logical operator does not short circuit (it evaluates both sides of the expression even if the first is False
). Before splitting the IRONPYTHONPATH
environment variable, we need to check that it is not null and not empty. We need to use the AndAlso
operator because we can’t check the length of the string if it is null.
Now we’re ready to execute the code, but with one caveat: because we are executing code in a dynamic language, we need to be able to handle runtime exceptions gracefully.
Our .NET code is just a thin wrapper around the Python program, so unhandled exceptions will be fatal anyway, and we could simply let them bubble up. The problem with this approach is that the traceback written to the console will be an incomprehensible CLR traceback, including all the stack frames inside the DLR. What we want is a straightforward Python traceback, and we can get this by catching the exception and using ExceptionOperations
to format the traceback.
We have to confess to slightly misleading you earlier; although Execute
is a perfectly valid way to run the code in a ScriptSource
, it isn’t the best way to run a script that is a full program. Python programs can exit with a return code[6] (an integer), and if this happens we want to propagate the exit code. Instead of Execute
, we can use ExecuteProgram
, which returns us the integer exit code (and will create its own scope rather than requiring us to pass one in). Listing 15.4 executes the ScriptSource
we created from program.py inside some exception-handling code. If the program terminates normally, it exits with the return code from ExecuteProgram
. If an exception is raised, it writes out the formatted exception message and exits with a return code of 1.
Example 15.4. Executing the ScriptSource
, handling any exceptions (in C#)
try { ScriptSource source; source = engine.CreateScriptSourceFromFile(programPath); int result = source.ExecuteProgram(); return result; } catch (Exception e) { ExceptionOperations eo = engine.GetService<ExceptionOperations>(); Console.Write(eo.FormatException(e)); return 1; }
We have pulled all this together into the EmbeddedExecutable
example in the downloadable source code.
Something the example does that isn’t shown in the code snippets above is to set debug mode on the engine. This is done by passing in a dictionary of options to the CreateEngine
call, with Debug
set to true
. It causes the IronPython engine to compile Python code in debug mode (without optimizations—so it isn’t recommended for production code). This allows you to use the Visual Studio debugger, including setting breakpoints, on the Python code.
When you run EmbeddedExecutable.exe, it runs program.py, which prints a message along with the command-line arguments it is passed. It then raises an exception to illustrate the exception formatting, as you can see in figure 15.2.
Embedding IronPython in a custom executable uses only a fraction of the hosting API, but we have used some of the most important classes. The ScriptEngine
, ScriptScope
, and ScriptSource
will accompany us through the rest of the book.
Next we look at some different hosting scenarios and explore more of the API. These build on what we have already learned, but instead of just executing Python code, we actually interact between the host application and the Python environment it is hosting.
By embedding IronPython in an application we can do much more than create executable wrappers over Python applications. Potential uses include providing scripting capabilities and plugins for .NET applications and even writing (or prototyping) parts of an application in IronPython.
In this section we work through another example,[7] which covers the core classes and basic techniques for embedding IronPython. In the next section we build on this with a more specific example that uses IronPython to add a plugin mechanism to a program.
Topics we cover in this section are creating compiled code objects from Python code, setting and fetching variables from execution scopes, adding references to assemblies, and publishing Python modules to runtimes. This will give us several different ways to interact between IronPython code and the .NET application. We can directly place objects into an execution context (a scope) for Python code to use—possibly calling back into our application from these objects. We can then fetch objects back out of the execution context after Python code has run. Alternatively, we can add references to assemblies or build and publish Python modules, so that IronPython code can access them by importing them.
The methods that initialize the main execution scope and then execute the Python code are very similar to the code we have already written. There are two important differences, though.
When we were running a Python program, we created a ScriptSource
from the source code file and called ExecuteProgram
to run it. ExecuteProgram
executes code in its own execution scope. However, if we execute code in an explicitly created ScriptScope
, we can then access variables, classes, and the like that have been created by the executed code. Conversely, if we want to provide objects for the code to use, we can place them in the scope prior to execution.
The second difference in the embedding code we are about to write is the way that code is executed. In the snippets we have just seen, a ScriptSource
is created from the source code and then executed directly with the scope. If we are going to execute the code several times, which is entirely likely in a hosting situation, then we can optimize by compiling the code from the ScriptSource
.
Calling script.Compile()
returns us a CompiledCode
object that we can use in place of the ScriptSource
.
ScriptScope
and CompiledCode
are two highly reusable components. You can create multiple scopes and execute the same code in the different scopes or create a single scope and execute different code with access to the same objects.
The ScriptScope
is a DLR class and is not tied to IronPython. You can execute Python code to create an object graph in a scope and then execute Ruby code in the same scope and with access to the same objects.[8] This permits some very interesting interoperability stories. Where IronRuby (or Managed JScript or IronScheme or ...) uses IronPython objects, they retain their behavior as Python objects but are still usable from these other languages. There are some restrictions in the ways they can interoperate; it is unlikely that Python classes will ever be able to inherit from Ruby classes or vice versa, for example.[9] It should still be possible for dynamic languages running on .NET to share libraries, though. Python on Rails or Ruby on Django, anyone?
From a hosting point of view, the interesting thing we can do with scopes is to set variables in and fetch variables out of them. This means that you can publish an object model from the hosting application into the scope for user code to work with (call methods, add handlers to events, and so on). After executing Python code you can then fetch objects it has created out of the scope.
The methods for working with names contained in a scope are shown in table 15.2.
Table 15.2. ScriptScope
methods for working with names and variables
Method name | Purpose |
---|---|
| Clears all the variables in the scope. |
| Returns |
| Fetches the named variable from the scope as Object. Will raise an UnboundNameException if the variable does not exist. |
| Fetches the named variable as type T. Will raise an ArgumentTypeException if the variable is of the wrong type. |
| Removes the named variable, returning |
| Sets the specified variable in the scope. |
| Takes an out parameter, which will not be set if the variable does not exist in the scope. |
You also have read-only access to all the variable names as IEnumerable
of string through the public VariableNames
property, and the variables themselves as IEnumerable
of KeyValuePair
s (containing string, object pairs) through the Items
property.
The most important of these methods are the three that allow you to set and fetch variables in the scope:
GetVariable
TryGetVariable
SetVariable
Setting variables is straightforward. You call SetVariable
with any string and any object. If you want Python code to actually have access to the variables you set, then you should restrict the strings (variable names) to be valid Python identifiers, but the scope doesn’t enforce this.[10]
Fetching variables has more complications associated with it, mainly because of the impedance mismatch of interacting with a dynamic language from a statically typed language. So long as the variable exists—and after executing arbitrary Python code there’s no guarantee of that—C# and VB.NET insist on knowing the type before they will allow you to do anything useful with it.
For known types you have a choice of checking that the variable exists with ContainsName
and use the generic version of GetVariable
to fetch it from the scope. To fetch a string you use GetVariable<string>(name)
from C# or GetVariable(OfString)(name)
from VB.NET. Alternatively you can use TryGetVariable
, which takes an out parameter that will be null (nothing) after the call if the variable doesn’t exist.[11] From C# you will then need to cast the value to the known type after fetching it. TryGetVariable
returns a Boolean indicating success or failure of attempting to fetch the variable.
If you are executing arbitrary code you will, therefore, need code that can handle the variable not existing or being the wrong type. If you are executing known code rather than arbitrary code, then it is fine to do any necessary error handling within the Python code and be able to guarantee that the variable exists and is of the expected type.
For .NET types, once you’ve pulled them out of the scope you have full access to them as if they were created from C#/VB.NET. If they are dynamic objects, such as functions or classes, you can still use them—but you have to use mechanisms provided by the Dynamic Language Runtime to perform operations on them.[12] This is something that we will look at later in the chapter.
In the meantime we now have all the pieces we need to set variables in a scope, execute code in that scope, and then fetch objects back out. Listing 15.5 is the start of an Engine
class in C#. You instantiate it with Python source code as a string.
The Execute
method catches any exceptions raised in the Python code (writing the Python exception to standard out) and returns a Boolean indicating success or failure. The SetVariable
and TryGetVariable
methods on the engine merely delegate to methods on the ScriptScope
. Handling error conditions, such as the variable not existing or being of the wrong type, is up to the caller.
This code does something that our last example didn’t. Now that we know how to set variables in our execution scope, we can set the __name__
. Python code would expect __name__
to exist, and for the top-level script it would expect it to be set to __main__
. The reason why we didn’t do this when creating the executable is that engine.CreateScriptSourceFromFile
implicitly sets the name in the scope to the name of the file (minus the .py).[13] The advantage of using CreateScriptSourceFromFile
is that it doesn’t read the whole file at once. If your top-level program depends on the name being set to __main__
, then you can have the best of both worlds by using the three-argument form: engine.CreateScriptSourceFromFile(path, Encoding.Default, SourceCodeKind.Statements)
.
This allows you to execute the ScriptSource
in a scope with an explicit __name__
set, without it being implicitly overridden because it is from a file.
With what you know already you could probably achieve almost everything you need when embedding IronPython. Magically injecting variables is not (always) the most elegant way of exposing objects to your hosted code. A much nicer solution is to make your objects available either in Python modules or in assemblies that the Python code can import in the usual way.
Adding references to assemblies is done with the LoadAssembly
method on the ScriptRuntime
. This takes an actual assembly object, so we need to use the System.Reflection
API to obtain it.
We can use this to overcome another minor limitation when embedding IronPython. ipy.exe adds references to mscorlib.dll and System.dll (the core assemblies containing the System
namespace) for us. This means that import System
, or variants, can be executed without explicitly having to add references to these assemblies. Code running inside the default embedded IronPython engine will need to manually add the references before being able to import from System
. We can solve this by getting Assembly objects from types in each of the two assemblies. We can do this in C# with this:[14]
_runtime.LoadAssembly(typeof(String).Assembly); _runtime.LoadAssembly(typeof(Uri).Assembly);
Which translates to this in VB.NET:
_runtime.LoadAssembly(GetType(String).Assembly) _runtime.LoadAssembly(GetType(Uri).Assembly)
The BasicEmbedding
example also includes a ClassLibrary.dll assembly containing a class with a couple of static methods (shared functions in VB.NET–speak) that write to stdout. We can add a reference to this assembly by first loading it with Assembly.LoadFile
from the same directory as the executable. This time we have the example code in VB.NET, shown in listing 15.6.
Once we’ve added a reference to the assembly, Python code running in the hosted engine is free to import from its namespace(s).
If the object model you want to expose isn’t contained conveniently in a single assembly, or you want to construct it from live objects at runtime, then an alternative is to construct a Python module and add that to the runtime instead.
In Python, import statements first check to see if the requested module is already in sys.modules
. The embedded equivalent is runtime.Globals
, and just like sys.modules
it doesn’t have to be a “real” Python module you put in there but any object you want Python code to be able to import.
ScriptRuntime.Globals
is actually our old friend the ScriptScope
. This means that you set objects in it using the same SetVariable
method we have already used. You can put any object into there, and import statements executed in the embedded engine will fetch them. Using this technique is another way to make a host object model available to user code.
The challenge is that we do want to create a real Python module to put in there. We could just create and populate a new ScriptScope
and publish that. Under the hood IronPython modules use a ScriptScope
to store the namespace, and it would behave like a module when imported. However, ScriptScope
doesn’t have the right repr
, and it has a few other minor differences, so it will seem a bit odd if the user does introspection on a ScriptScope
published as a module. The answer is to use the Scope
object, which as far as IronPython is concerned is a real Python module. We construct a Scope
object from a ScriptScope
using HostingHelpers.GetScope
.[15]
The following snippet of VB.NET creates a ScriptScope
called inner
in which we set a string with the name HelloWorld
. This is wrapped in a Scope
object that is then published into the runtime globals.
Dim _module As Scope Dim inner As ScriptScope inner = _engine.CreateScope() inner.SetVariable("HelloWorld", "Some string...") _module = HostingHelpers.GetScope(inner) _runtime.Globals.SetVariable("Example", _module)
Code running in the embedded engine can either execute import Example
and access HelloWorld
as a module attribute or execute from Example import HelloWorld
to get direct access to the string we set in inner
.
We’ve now covered all the major classes necessary for a wide range of different embedding scenarios. Figure 15.3 summarizes what we learned so far. It shows the core classes that we have worked with and the relationship between them and their useful members. Our core Engine
class is now basically complete, with only one minor modification needed. Since we know how to make modules available for importing, we turn the scope in which we execute the main script into a proper module.
This C# snippet does this and puts the module into the runtime globals with the name __main__
:
_scope = _engine.CreateScope(); _scope.SetVariable("__name__", "__main__"); Scope _main = HostingHelpers.GetScope(_scope); _runtime.Globals.SetVariable("__main__", _main);
The advantage of doing this is that the Python code import __main__
now does the right thing. This is not a common thing to do but is used, for example, by unittest.main()
to introspect the main execution scope and find all TestCase
classes.
The example project uses the Engine
class by embedding a Python script as a resource. Let’s see how it uses the engine.
This example executes a Python file that is stored as an embedded resource compiled into the main assembly. This can be a useful way of preventing Python code your application depends on from being modified in your distributed application. Even though it isn’t distributed as plain text, it isn’t an effective way of keeping the source code secret, as it will be easily discoverable within the assembly.[17]
Adding a Python source file as an embedded resource to a Visual Studio project is as simple as adding the file (either from an existing file or adding a new text file and renaming) and setting the Build Action to Embedded Resource, as shown in figure 15.4.
Listing 15.7 shows the C# code to retrieve the source code from the embedded resource as a string.
Example 15.7. Fetching an embedded resource from an assembly (in C#)
static string GetSourceCode() { Assembly assembly = Assembly.GetExecutingAssembly(); string name = "BasicEmbedding.source_code.py"; Stream stream = assembly.GetManifestResourceStream(name); StreamReader textStreamReader = new StreamReader(stream); return textStreamReader.ReadToEnd(); }
The important call here is assembly.GetManifestResourceStream(name)
, which gives us access to our embedded source file as a stream.
Our example retrieves the source code and uses the Engine
we have created to execute it. Before executing it sets a variable (imaginatively called variable) into the execution scope and fetches it out again after execution. Listing 15.8 shows the full code in VB.NET.
engine.Execute
does the error handling for us, returning a Boolean indicating whether or not the execution succeeded. Assuming we can’t trust our code not to fail, we need to check this result and handle the failure. Because we are actually executing code from an embedded resource and not arbitrary user code, we could skip the error checks if they aren’t necessary. If the code executes correctly, it fetches the variable
variable back out of the execution scope (handling the case when it doesn’t exist in the scope) and writes it out to the console. Because we are writing it out to the console, we fetch it as an object
, neatly bypassing the problem of having to know what type it is.
Unfortunately, this type problem we skipped over is quite a big issue for using embedded IronPython for anything practical. Although we already have most of the pieces of the jigsaw, the next section puts them together (and shows one way of resolving the type problem) by implementing a plugin system for a .NET application.
The example we’ve just worked through has taken us through a good proportion of the IronPython hosting API. There are still some useful tricks we can learn, though. In particular we can solve the problem of how to communicate between dynamic languages and C#/VB.NET, which have to know the types of objects in order to be able to do anything useful with them.
The goal of this example is to create a .NET application that allows the user to write plugins that extend the functionality in Python. The application interface consists of a toolbar and a multiline textbox.
The user can add plugins by creating Python files in the application’s plugins directory, which are all loaded when the application starts. Any plugin that is successfully loaded has a button added to the toolbar, and the plugin is called—with access to the application textbox—when its button is clicked. The final product is shown in action in figure 15.5.
Although this is a trivially simple example, it solves the real-world problems involved in creating a user-extensible application. It loads an arbitrary number of Python plugin scripts and provides them with an API in a clean manner. As well as bridging between code written in a dynamically typed language and an application written in statically typed languages, we will also be covering some useful techniques for plugin situations. These include
The autodiscovery of plugins, dynamically executing them at runtime rather than requiring configuration or other mechanisms
Diverting the standard output and error streams for hosted DLR runtimes and their engines
Handling specific exceptions from hosted Python code
We start by looking at how to create the host environment so that Python code can simply and cleanly add plugins.
In order to solve the type problem, we can provide a base class that plugin classes must inherit from. On the .NET side we can handle user plugins as this base class, and the compiler is happy—knowing what methods and properties are available.[18]
We also have to provide a mechanism for the plugins to be added to the application. We can do this with a PluginStore
class, also accessible to user code, which acts as a registry for plugins.
Our PluginBase
class is instantiated with a name, which will be used for the toolbar button. It also provides an Execute
method, which does nothing on the base class but will be called with the textbox in real plugins (when the corresponding toolbar button is clicked).
On the .NET side, where we need to interact with the user plugins, we can use the PluginBase
type. Listing 15.9 shows an implementation in C#.
The PluginStore
class is equally simple. It has a public static (shared) method, AddPlugin
, which takes a PluginBase
instance. This adds the plugin to a list accessed as the Plugins
property. This list is marked as internal (friend), which means that only classes in the same assembly can access the list and it isn’t exposed to user code. Listing 15.10 shows the PluginStore
class in VB.NET.
Example 15.10. The PluginStore
registry class (in VB.NET)
Public Class PluginStore Private Shared _plugins As List(Of PluginBase) _ = New List(Of PluginBase)() Friend Shared ReadOnly Property Plugins() _ As List(Of PluginBase) Get Return _plugins End Get End Property Public Shared Sub AddPlugin(ByVal plugin As PluginBase) _plugins.Add(plugin) End Sub End Class
In fact, the PluginStore
is the only public class in the main assembly. We can add the main assembly to the runtime (along with the assembly containing the PluginBase
), and user code can call PluginStore.AddPlugin
without being able to access any of the application internals that we haven’t explicitly exposed.
Listing 15.11 shows the Python code that creates a new plugin and adds it to the PluginStore
.
Example 15.11. IronPython plugin using PluginBase
and PluginStore
from Plugins import PluginBase from EmbeddingPlugin import PluginStore class Plugin(PluginBase): def Execute(self, textbox): textbox.Text += "Plugin 1 called " plugin = Plugin("Plugin 1") PluginStore.AddPlugin(plugin)
Our application can execute all the user scripts and then access any successfully added plugins via PluginStore.Plugins
. Let’s look at how our application loads the user scripts.
Our application uses a very simple mechanism to load user scripts. Alongside the executable is a directory called plugins. When the application starts, it executes all the Python files in this directory.
As we are executing user code we obviously need to be tolerant of errors, and to make things more interesting, let’s see how we can handle specific Python errors differently.
The SyntaxErrorException
is raised when we attempt to execute invalid code. It lives in the Microsoft.Scripting
namespace. It is an exception shared by all DLR languages. The Python-specific exceptions live in the IronPython.Runtime.Exceptions
namespace, so with the right using
or Imports
directive we can catch specific Python errors. Listing 15.12 shows the code that loads and executes all the plugins. Any errors in executing plugin code are caught, and a message box is displayed to the user, but syntax errors and SystemExit
exceptions[19] are treated differently with a custom message. Figure 15.6 shows the error message shown to users for syntax errors.
Listing 15.12 is the C# code that loads and executes the scripts from the plugins directory.
The ShowError
method may seem slightly odd: the caller has to extract the filename from the full path instead of it being done inside the method. It is written this way because we reuse ShowError
to show a different kind of message when we call the plugins.
Before we finish off the application by hooking up the plugins to the toolbar buttons, we look at how we can improve the development process for the user writing the plugins.
This example is a Windows Forms application, which makes for much better screen-shots. It does have one disadvantage, though; it means that print statements inside the user code are lost because there is no standard output for them to go to. This could make debugging plugins much harder for our users. Luckily, DLR runtimes support diverting the standard output and error streams, and in this example we send them to the application textbox.
The API for diverting the output and error streams[20] is through runtime.IO.SetOutput
and runtime.IO.SetError
methods that take a .NET Stream
and an encoding. To use these methods, we need a Stream
that diverts everything written to it back to the textbox.
Stream
is an abstract class—inheriting from it is easy but requires implementing a tedious number of methods and properties. For this example we’ve chosen to inherit from MemoryStream
and override the Write
method. This is an abuse of MemoryStream
, but it works fine.
Listing 15.13 shows the PythonStream
class and setting an instance onto the runtime to divert both standard output and standard error.
Example 15.13. Diverting output streams to a textbox (in C#)
internal class PythonStream: MemoryStream { TextBox _output; public PythonStream(TextBox textbox) { _output = textbox; } public override void Write(byte[] buffer, int offset, int count) { string text = Encoding.UTF8.GetString(buffer, offset, count); _output.AppendText(text); } } public void SetStreams() { PythonStream stream = new PythonStream(_box); _runtime.IO.SetOutput(stream, Encoding.UTF8); _runtime.IO.SetErrorOutput(stream, Encoding.UTF8); }
Minor points are worth noticing from this listing. When we set the streams on the runtime we need to specify an encoding; so in our custom stream we need to use the same encoding to decode the bytes back into a string. The stream is constructed with a textbox, and the Write
method simply calls AppendText
to add new text.
The “Loading Plugin 1” and other messages shown in figure 15.5 actually come from print statements in the plugin code. When the plugin is loaded (executed), anything written to standard out appears in the main textbox.
The final thing we need to do is to create a toolbar button per plugin and hook up its Click
event to call the appropriate Execute
method.
Once the plugins have all been loaded, we can iterate over PluginStore.Plugins
and add a button and handler for each one. The code to do this from C# is shown in listing 15.14.
Example 15.14. Adding toolbar buttons and click handlers per plugin (in C#)
int index = 0; foreach (PluginBase plugin in PluginStore.Plugins) { ToolStripButton button = new ToolStripButton(); button.ToolTipText = plugin.Name; button.Text = plugin.Name; int pluginIndex = index; button.Click += delegate { ExecutePluginAtIndex(pluginIndex); }; pluginToolStrip.Items.Add(button); index++; } public void ExecutePluginAtIndex(int index) { PluginBase plugin = PluginStore.Plugins[index]; try { plugin.Execute(_textbox); } catch (Exception e) { string msg = "Error executing plugin "{0}""; ShowError(msg, plugin.Name, e); } }
In C# we can use a closure to call the right plugin when its handler is invoked. By turning the index into a variable local to the scope of the foreach
loop (the pluginIndex
variable), the anonymous delegate will invoke ExecutePluginAtIndex
with the correct index when its button is clicked.
The VB.NET code, shown in listing 15.15, has to be slightly different.
Example 15.15. Adding toolbar buttons and click handlers per plugin (in VB.NET)
Dim index As Integer = 0 For Each plugin As PluginBase In PluginStore.Plugins Dim button As ToolStripButton = New ToolStripButton() button.ToolTipText = plugin.Name button.Text = plugin.Name Dim handler As ButtonHandler handler = New ButtonHandler(engine, index, button) pluginToolStrip.Items.Add(button) index += 1 Next Friend Class ButtonHandler Dim _engine As Engine Dim _index As Integer Public Sub New(ByVal engine As Engine, _ ByVal index As Integer, _ ByVal button As ToolStripButton) _engine = engine _index = index AddHandler button.Click, AddressOf ClickHandler End Sub Public Sub ClickHandler(ByVal sender As Object, _ ByVal e As EventArgs) _engine.ExecutePluginAtIndex(_index) End Sub End Class Public Sub ExecutePluginAtIndex(ByVal index As Integer) Dim plugin As PluginBase = PluginStore.Plugins(index) Try plugin.Execute(_box) Catch e As Exception Dim msg As String = "Error executing plugin ""{0}""" ShowError(msg, plugin.Name, e) End Try End Sub
Because we don’t have the luxury of closures in VB.NET, we have to use a class to capture the state (which is what the compiler does under the hood for us in C# anyway). For every plugin a new ButtonHandler
instance is created, which has a ClickHandler
to be called when the button is used.
In the previous two sections of this chapter we have gone through two different examples of embedding IronPython. The first was a gentle amble through the hosting API that introduced us to all the major classes. The example that we have just completed showed us how to use embedded IronPython for a practical purpose, in the form of application plugins. This example solved the mismatch between dynamic and statically typed languages by using a specific type for plugins. We can also use more general techniques to interact with dynamic objects and, in the process, extend IronPython to some new use cases.
One of the (many) great advantages of Python is that code can be modified without having to be recompiled. It is stored as plain text and can even be generated at runtime. There has been a lot of focus in the technical world recently on using dynamic languages to provide Domain Specific Languages (DSLs), small languages that model a particular problem domain.
We can use IronPython for this purpose by evaluating expressions or executing code stored as text or received as user input. Business rules can be stored in text files and edited without the application needing to be recompiled.
This use case requires new ways of interacting with dynamic objects. We need to be able to create and use normal Python objects (the built-in types and Python classes and instances) from .NET, while retaining their behavior as Python objects.
The simplest example of this is to use IronPython to evaluate individual expressions. IronPython makes a great calculator![21]
So far, whenever we have executed Python code from a string we have passed in the enumeration member SourceCodeKind.Statements
. This member has a sister, SourceCodeKind.Expression
, that allows us to evaluate an expression and return an object. It is used in this snippet of C# to evaluate a simple mathematical expression:
string code = "2 + 3 + 5"; ScriptSource source; source = runtime.CreateScriptSourceFromString(code, SourceCodeKind.Expression); int result = source.Execute<int>(scope);
This code uses the generic form of Execute
that allows us to specify the type of the returned object. Using the generic version requires us to pass in an explicit scope to Execute
. If we wanted to use the default scope, we could instead cast the returned object to retrieve the integer.
Since we are retrieving results with specific types, we can use the System.Func
delegate to define functions in Python and call them from other .NET languages. This delegate is a standard part of .NET 3.5 (C# 3.0), but it is also provided by IronPython 2, so you can happily use it in .NET 2.0 projects.
The simplest example of this is to evaluate an expression that returns us a lambda function, as shown in listing 15.16.
Example 15.16. Creating and using Python functions from C#
string code = "lambda x, y: x * y"; ScriptSource source; source = engine.CreateScriptSourceFromString(code, SourceCodeKind.Expression); Func<int, int, int> lambda; lambda = source.Execute<Func<int, int, int>>(scope); int result = lambda(9, 8); Console.WriteLine("9 * 8 = {0}", result);
Listing 15.17 shows the equivalent VB.NET code.
Example 15.17. Creating and using Python functions from VB.NET
Dim code As String = "lambda x, y: x * y" Dim source As ScriptSource Dim lambda As Func(Of Integer, Integer, Integer) source = engine.CreateScriptSourceFromString(code, SourceCodeKind.Expression) lambda = source.Execute( _ Of Func(Of Integer, Integer, Integer))(scope) Dim result As Integer = lambda(9, 8) Console.WriteLine("9 * 8 = {0}", result)
Func
is generic, with type specifiers for the arguments and the return value. This example specifies a function that takes two integers as arguments and returns an integer: Func<int
, int
, int>
. The last type specified is always the return type. There are convenient definitions for functions that take up to eight arguments (and if that isn’t enough, you need to rethink your function API!).
Instead of evaluating an expression to create a lambda, we could execute a function definition in a scope and pull the function out by name. This would work with any callable object, and we could pull them out as any form of delegate—allowing the ScriptScope
to do the language conversion when we pull out the value.
As well as allowing you to use dynamically created functions, you could use Python functions to provide a .NET API to Python libraries.
We can use functions because they have a corresponding type defined in C#, and the same is true of the other Python types. The built-in Python types live in the IronPython.Runtime
namespace. The very basic types, like strings and integers, aren’t there because IronPython uses the standard .NET types for these. Types that are available include the following:
List
FrozenSetCollection
PythonDictionary
PythonFile
PythonGenerator
PythonTuple
SetCollection
Most of the normal Python methods are available on these types, including the magic double-underscore methods. You can explore the types and their members using Visual Studio 2008 Object Browser or Reflector.[23] Figure 15.7 shows some of the members on the Python List type.
We can retrieve these objects by type from a Python scope or create and use them directly. When we created a custom executable for a Python program, we created a Python List and set it on the language context as sys.argv
. Listing 15.18 shows creating a tuple in Python and then using it from C#.
Example 15.18. Creating a tuple in Python and using it from C#
string code = "('hello', 'world')"; ScriptSource source; source = engine.CreateScriptSourceFromString(code, SourceCodeKind.Expression); PythonTuple tuple = source.Execute<PythonTuple>(scope); Console.WriteLine("tuple = ({0}, {1})", tuple[0], tuple[1]);
These techniques are great for the built-in Python types, and for objects with known types, but they don’t allow us to use classes defined in Python. Fortunately, there are other techniques we can use with Python classes and instances.
IronPython (and IronRuby) classes are not .NET classes. IronPython classes are .NET objects—instances of IronPython.Runtime.Types.PythonType
, the Python type
metaclass.[24] This is because Python classes can have members added and removed at runtime. You can even change their base classes dynamically, things that we couldn’t do with .NET classes.
Python classes are still usable from C# and VB.NET; we can actually solve the type problem by ignoring it! We can keep objects as objects on the .NET side and use the Python engine to perform operations like creating instances and calling methods.
The mechanism for doing this is to use ObjectOperations
, a class that provides dynamic operations for DLR objects. The ScriptEngine
exposes this as the Operations
property, returning an ObjectOperations
instance bound to the semantics of the engine’s language.
ObjectOperations
is a class with many useful methods, and again the best way to find out what it can do is to explore it with Reflector or the Visual Studio Object Browser. It knows how to perform comparisons and mathematical operations on dynamic objects, following the semantics of the language for those operations. In the case of Python that means autopromoting integers to longs if necessary and using the __add__
method for addition where appropriate, and so on.
More important, ObjectOperations
knows how to call objects and set and fetch members. Using these capabilities alone we can achieve most of what we might want to do with dynamic objects. Listing 15.19 shows how to fetch a Python class out of an execution scope, create an instance of that class, and call a method on it.
Calling the method is a little cumbersome, but if we were working with known Python classes, it would be easy to abstract away with a thin wrapper layer.
We create an instance of a class in the same way we do in Python, by calling the class. This is done with ops.Call
, and afterward we fetch a method from the instance with ops.GetMember
. GetMember
takes the instance and the name of the member we want to fetch as a string. This mirrors the way we use the getattr
function from Python.
Having fetched the method we call it, again with ops.Call
. Any arguments passed to ops.Call
are passed in as arguments to the object we are calling; the overload we are using is Call(object, params object[])
. We cast the objects to known types only at the point at which we actually need them, in this case the return value from calling the method. GetMember
also has a generic version, which we can call if we care about the type of object it will return.
The corresponding partner to GetMember
is SetMember
. Its signature matches setattr; ops.SetMember(object, name, value)
where name
is a string specifying the member to set.
Table 15.3 lists methods of ObjectOperations
, including the three we have already discussed.
Table 15.3. Methods on ObjectOperations
for working with dynamic objects
Method name | Purpose |
---|---|
| Calls the object with any arguments specified |
| The equivalent of the two-argument form of |
| The equivalent of |
| The equivalent of |
| Lists all members; the equivalent of |
| The equivalent of |
| Takes an out parameter that will be left unmodified if the member does not exist |
| Converts a dynamic object to the specified type, either in generic form or taking a |
It may not be immediately obvious why we might need ConvertTo
and TryConvertTo
when we can always cast objects. We can use them to take advantage of some of the magic that IronPython does on our behalf, particularly when using Python functions as event handlers. Under the hood IronPython wraps the function as a delegate for us, and we can use ObjectOperations.ConvertTo
to do the same from the .NET side. The following snippet shows how to do this from C#:
ObjectOperations ops = engine.Operations; object function = scope.GetVariable("function"); SomeObj.SomeEvent += ops.ConvertTo<DelegateType>(function);
Remember that if you want to be able to unhook the handler, you’ll need to keep a reference to the delegate it returns!
As well as being able to fetch and call methods on dynamic objects, we can use the Python built-in functions on them.
If we have a collection of objects created from Python code and we want to add them up, we have several choices of how to do it. We could set them in an execution scope with names and evaluate an expression that adds them up, or we could use ObjectOperations.Add
in a loop. We have a third choice as well; Python has a perfectly good built-in function for adding things up: sum
.
Python has a wide range of useful built-in functions, and in IronPython they are implemented as static methods on IronPython.Runtime.Builtin
. Some of the builtin functions require a CodeContext
; we’ll see how to provide this when we look at working with built-in modules. sum
is perhaps not the most useful example to pick, but some of the built-in functions are very useful for working with dynamic objects.
For example, if we have obtained (by whatever means) a dynamic object that we believe to be an instance of a particular class, we can confirm this by using the isinstance
function. Listing 15.20 shows an example of using both the isinstance
and the issubclass
functions from C#.
This example creates two Python classes and fetches them both out of the scope. It creates an instance of the first one (Something
) and then performs various checks on the instance and the classes using Builtin.isinstance
and Builtin.issubclass
.
One difference about the way we use Python classes in this example is that we are specifically pulling them out as PythonType
rather than object
. isinstance
takes two objects, but issubclass
requires the first argument to be a PythonTyp
e (new style class) or OldClass
(guess).
We’ve now used the built-in types and the built-in functions, but there is one more class of Python built-in that we haven’t used directly with embedded IronPython. From the title of this section it won’t come as a shock to you to hear that these are the built-in Python modules.
The built-in modules are implemented in the IronPython.Modules
assembly. In there you will find a single entry for each module (ClrModule
, PythonDateTime
, PythonSocket
, and so on) plus one for each type that the module exports. Functions in the module are defined as static methods on the module object.
Listing 15.21 uses the Python pickle
module[25] to serialize and deserialize a Python dictionary. It then checks that the serialization and deserialization have worked, using ObjectOperations.Equal
to compare the original dictionary and the deserialized object. In order to use the pickle loads
and dumps
functions, we must first create a CodeContext
.[26]
The call to PythonPickle.dumps
takes a CodeContext
, a Python object, a protocol, and a bin
argument. First we construct a CodeContext
with a new Scope
and the appropriate LanguageContext
. Creating CodeContext
s like this is useful for working with several of the built-in functions and modules.
This example uses protocol 0, which ensures that the resulting pickle
is serialized as ASCII rather than using a binary format. Storing binary data in Unicode .NET strings is a recipe for pain, and it means we can pass in null
for the bin
argument.
dumps
returns an object, which we can cast to a string. If we feed this string back into PythonPickle.loads
(along with the context), it deserializes the string back into a Python dictionary for us. This could be useful for persisting state from the dynamic part of an application.
The DLR hosting API is larger than we could hope to cover in a single chapter. Topics we haven’t had a chance to look at include the Dynamic Language Runtime support for creating engines and executing code inside AppDomains. This is a good way of limiting what user code is able to do when run inside your application. It also has a Platform Adaptation Layer (PAL), used by Silverlight, that allows you to customize the way files are opened and imports are resolved and even to execute code in custom namespaces that can provide or change names on demand.
Not only that, but there is an enormous number of ways you could use embedded IronPython. What we have managed to do, though, is look at some of the most common ways of using IronPython, and most of the numerous alternatives will bear some similarity to the scenarios we have explored. What I really hope you take away from this chapter is a kick-start in being able to imagine what is possible and knowing how to start exploring. Before we finish, there is some big news about how you will interact with dynamic languages in future versions of the .NET framework.
At the 2008 Microsoft PDC conference, Jim Hugunin and Anders Hejlsberg announced[27] that the DLR would be integrated into version 4.0 of the .NET framework. Alongside this there will be changes to C# and Visual Basic to introduce dynamic features that use the DLR.
The major change in C# 4.0, at least the one that is relevant to us, is the addition of the dynamic
keyword. This is a static declaration to the compiler that operations on the object are to be handled dynamically at runtime! Operations on objects declared as dynamic will be delegated to the DLR. For ordinary .NET objects the DLR uses reflection (just as it does inside IronPython), but it also enables some things not normally possible from C#. These include duck typing, the use of late-bound COM, and the creation of fluent APIs,[28] like XML or DOM traversal using element names as object attributes.
More important, it allows you to receive objects from a DLR language engine and use them as dynamic objects. Objects created by IronPython or IronRuby can be used from C# while retaining their behavior as Python and Ruby objects.
The C# Future documentation[29] gives this example of using the new dynamic
keyword. All of the uses of d shown here will be done by the DLR:
dynamic d = GetDynamicObject(...); d.M(7); // calling methods d.f = d.P; // getting and settings fields and properties d["one"] = d["two"]; // getting and setting through indexers int i = d + 3; // calling operators string s = d(5,7); // invoking as a delegate int a = d; // assignment conversion
The final operation creates a typed object (a
) from the dynamic object d
. As with the examples you’ve been working on in this chapter, these operations could raise runtime errors, and so the sort of error handling that we’ve been discussing will still be needed.
The bottom line is that the hosting APIs make it easy to work with dynamic languages, but interacting with dynamic objects will get a whole lot easier.
The integration of IronPython with the underlying .NET runtime is what makes it such a joy to work with. This extends to embedding IronPython in C# and VB.NET. Creating and using language engines is straightforward (once you know the magic incantations), so experimenting is easy and enables things that previously might have required writing your own scripting language!
There is a natural complication when using objects created by a dynamic language from statically typed languages; this is something that can and will[30] get easier, but there are many ways you can minimize the intricacy. One useful principle is to do as much as possible inside the engine. If you do your type checking and error handling in Python, then you can guarantee to return a known type into your statically typed code. When you really want to work with a dynamic object, then ObjectOperations
is your friend.
We’ve now used IronPython from both the inside and the outside, and we’ve also reached the end of the last chapter. This willingness to experiment, and an excitement about the possibilities, is the most important message of the book. Have fun programming!
[1] Domain Specific Languages—These are “little languages” that encapsulate rules for a specific problem domain.
[3] The IronRuby project has a similar entry point with easy access to engines and runtimes preconfigured for working with IronRuby.
[4] There is also support for creating separate runtimes in different AppDomains for further isolation.
[5] This is available from the DLR project page on CodePlex: http://www.codeplex.com/dlr.
[6] A Python program exits with an explicit return code by calling sys.exit(integer)
.
[7] We use the BasicEmbedding
example in the downloadable sources.
[8] The Silverlight DLRConsole application does exactly this.
[9] If they could, which metaclass should they use—the Python one or the Ruby one?
[10] This mirrors the behavior of Python, where you can also programmatically set invalid identifiers in a namespace.
[11] Or if the variable is set to None
inside the scope.
[12] This is not hard, but will be even easier once support for dynamic operations is built into the CLR or C#/VB.NET languages—as is happening in C# 4.0 and VB.NET 10.
[13] This is the same behavior when importing a module in Python; the name in the module corresponding to the file os.py is os
, for example.
[14] The choice of Uri is entirely arbitrary. We just need some class that lives in System.dll.
[15] HostingHelpers
lives in the Microsoft.Scripting.Hosting.Providers
namespace. ScriptScope
s are “remotable wrappers” for Scope
s. HostingHelpers.GetScope
gets the local version, so it will work only for local ScriptScope
s. There are other ways of creating Scope
s when using the DLR remoting support.
[16] Some of these classes have other useful members. This diagram is a reference to the ones we have used so far.
[17] Although you could encrypt your Python source code. Because that will require the means to decrypt the code in the assembly as well, it is still discoverable—but probably more work than assemblies compiled from C#, which are usually trivially disassembled with Reflector.
[18] Additional user-defined methods and members won’t be directly visible, of course.
[19] This is raised if the user calls sys.exit(n)
.
[20] We could also do this from within Python code by replacing sys.stdout
and sys.stderr
with custom objects.
[21] You can see an example of embedded IronPython as a calculator at http://www.voidspace.org.uk/ironpython/dlr_hosting.shtml.
[22] Actually it has two more. The purpose of SourceCodeKind.SingleStatement
speaks for itself.
[23] This extremely useful tool for introspecting .NET assemblies is available at http://www.red-gate.com/products/reflector/.
[24] All Python classes are instances of their metaclass, and for new style classes the base metaclass is type
. type
can also be used as a function to tell you the type of objects.
[26] CodeContext
lives in the Microsoft.Scripting.Runtime
namespace.
[27] Jim’s blog has links to videos of their talks: http://blogs.msdn.com/hugunin/archive/2008/10/29/dynamic-language-runtime-talk-at-pdc.aspx.
[28] .NET objects that implement the IDynamicObject
interface can provide custom behavior when used dynamically.
[29] This documentation is available from http://code.msdn.microsoft.com/csharpfuture/.
[30] For example, see the discussion about C# 4.0 at http://channel9.msdn.com/posts/Charles/C-40-Meet-the-Design-Team/.
3.22.71.28