Defining the Common Language Infrastructure

Instead of generating instructions that a processor can interpret directly, the C# compiler generates instructions in an intermediate language, the Common Intermediate Language (CIL). A second compilation step then occurs, generally at execution time, that converts the CIL to machine code that the processor can understand. Conversion to machine code is still not sufficient for code execution, however. It is also necessary for a C# program to execute under the context of an agent. The agent responsible for managing the execution of a C# program is the Virtual Execution System (VES), more casually referred to as the runtime. (Note that the runtime in this context does not refer to a specific time, such as execution time; rather, the runtime—the VES—is an agent responsible for managing the execution of a C# program.) The runtime is responsible for loading and running programs and providing additional services (security, garbage collection, and so on) to the program as it executes.

The specifications for the CIL and the runtime are contained within an international standard known as the Common Language Infrastructure (CLI).¹ The CLI is a key specification for understanding the context in which a C# program executes and how it can seamlessly interact with other programs and libraries, even when they are written in other languages. Note that the CLI does not prescribe the implementation for the standard but rather identifies the requirements for how a CLI framework should behave once it conforms to the standard. This provides CLI implementers with the flexibility to innovate where necessary while still providing enough structure that programs created by one CLI implementation can execute on a different implementation and even on a different operating system.

1 Throughout the chapter, CLI refers to Common Language Infrastructure rather than command-line interface as in Dotnet CLI.

Contained within the CLI standard are specifications for the following:

• The Virtual Execution System

• The Common Intermediate Language

• The Common Type System

• The Common Language Specification

• Metadata

• The framework

This chapter broadens your view of C# to include the CLI, which is critical to how C# programs operate and interact with programs and with the operating system.

CLI Implementations

The primary implementations of the CLI today are .NET Core, which runs on Windows, as well as UNIX/Linux and Mac OS; the .NET Framework for Windows; and Xamarin, which is intended as a general cross-platform solution including iOS, macOS, and Android applications. Each implementation of the CLI includes a C# compiler and a set of framework class libraries. The version of C# supported by each, as well as the exact set of classes in the libraries, varies considerably, and many implementations are now only of historical interest. Table 24.1 describes these implementations.

Table 24.1: Implementations of the CLI

While the list is extensive considering the work required to implement the CLI, just three frameworks are the most relevant going forward.

.NET Standard

Historically, it has been quite difficult to write a library of C# code that can be used on multiple operating systems or even different .NET frameworks on the same operating system. The problem was that the framework APIs on each framework had different classes available (and/or methods in those classes). The .NET Standard solves this issue by defining a common set of .NET APIs that each framework must implement to be compliant with a specified version of the .NET Standard. This uniformity ensures that developers have a consistent set of APIs available to them across each .NET framework that is compliant with the .NET Standard version they target. If you wish to write your core application logic once and ensure that it can be used in any modern implementation of .NET, the easiest way to do so is to create a .NET Standard library project (available as a project type in Visual Studio 2017 and above or from the class library template with the Dotnet CLI). The .NET Core compiler will ensure that any code in the library references only APIs available to the version of the .NET Standard you target.

Class library authors should think carefully when choosing which standard to support. The higher the version number of the .NET Standard you target, the less you need to worry about writing your own implementation of APIs that might be missing in lower .NET Standard versions. However, the disadvantage of targeting a higher .NET Standard version is that it will not be as portable across different .NET frameworks. If you wish your library to work with .NET Core 1.0, for example, then you will need to target .NET Standard 1.6 and consequently will not have access to all the reflection APIs that are common to the Microsoft .NET Framework. To summarize this dichotomy: Target higher .NET Standard versions if you are lazy and lower .NET Standard versions if portability is more important than reducing your workload.

For more information, including the mapping of .NET framework implementations and their versions to their corresponding .NET Standard version, see http://itl.tc/NETStandard.

Base Class Library

In addition to providing a runtime environment in which CIL code can execute, the CLI defines a core set of class libraries that programs may employ, called the Base Class Library. The class libraries contained in the BCL provide foundational types and APIs, allowing programs to interact with the runtime and underlying operating system in a consistent manner. The BCL includes support for collections, simple file access, some security, fundamental data types (string, among others), streams, and the like.

Similarly, a Microsoft-specific library called the Framework Class Library (FCL) includes support for rich client user interfaces, web user interfaces, database access, distributed communication, and more.

C# Compilation to Machine Code

The HelloWorld program listing in Chapter 1 is obviously C# code, and you compiled it for execution using the C# compiler. However, the processor still cannot directly interpret compiled C# code. An additional compilation step is required to convert the result of C# compilation into machine code. Furthermore, the execution requires the involvement of an agent that adds more services to the C# program—services that it was not necessary to code for explicitly.

All computer languages define syntax and semantics for programming. Since languages such as C and C++ compile to machine code, the platform for these languages is the underlying operating system and machine instruction set, be it Microsoft Windows, Linux, macOS, or something else. In contrast, with languages such as C#, the underlying context is the runtime (or VES).

CIL is what the C# compiler produces after compiling. It is termed a common intermediate language because an additional step is required to transform the CIL into something that processors can understand. Figure 24.1 shows the process.

In other words, C# compilation requires two steps:

1. Conversion from C# to CIL by the C# compiler

2. Conversion from CIL to instructions that the processor can execute

The runtime understands CIL statements and compiles them to machine code. Generally, a component within the runtime performs this compilation from CIL to machine code. This component is the just-in-time (JIT) compiler, and jitting can occur when the program is installed or executed. Most CLI implementations favor execution-time compilation of the CIL, but the CLI does not specify when the compilation needs to occur. In fact, the CLI even allows the CIL to be interpreted rather than compiled, in the same way that many scripting languages work. In addition, .NET includes a tool called NGEN that enables compilation to machine code prior to running the program. This pre-execution-time compilation needs to take place on the computer on which the program will be executing because it will evaluate the machine characteristics (processor, memory, and so on) as part of its effort to generate more efficient code. The advantage of using NGEN at installation (or at any time prior to execution) is that you can reduce the need for the jitter to run at startup, thereby decreasing startup time.

As of Visual Studio 2015 and later, the C# compiler also supports .NET native compilation, whereby the C# code is compiled into native machine code when creating a deployed version of the application, much like using the NGEN tool. Universal Windows Applications make use of this feature.

Runtime

Even after the runtime converts the CIL code to machine code and starts to execute it, it continues to maintain control of the execution. The code that executes under the context of an agent such as the runtime is managed code, and the process of executing under control of the runtime is managed execution. The control over execution transfers to the data; this makes it managed data because memory for the data is automatically allocated and de-allocated by the runtime.

Somewhat inconsistently, the term Common Language Runtime is not technically a generic term that is part of the CLI. Rather, CLR is the Microsoft-specific implementation of the runtime for the .NET framework. Regardless, CLR is casually used as a generic term for runtime, and the technically accurate term, Virtual Execution System, is seldom used outside the context of the CLI specification.

Because an agent controls program execution, it is possible to inject additional services into a program, even though programmers did not explicitly code for them. Managed code, therefore, provides information to allow these services to be attached. Among other items, managed code enables the location of metadata about a type member, exception handling, access to security information, and the capability to walk the stack. The remainder of this section includes a description of some additional services made available via the runtime and managed execution. The CLI does not explicitly require all of them, but the established CLI frameworks have an implementation of each.

Assemblies, Manifests, and Modules

Included in the CLI is the specification of the CIL output from a source language compiler, usually an assembly. In addition to the CIL instructions themselves, an assembly includes a manifest that is made up of the following components:

• The types that an assembly defines and imports

• Version information about the assembly itself

• Additional files the assembly depends on

• Security permissions for the assembly

The manifest is essentially a header to the assembly, providing all the information about what an assembly is composed of, along with the information that uniquely identifies it.

Assemblies can be class libraries or the executables themselves, and one assembly can reference other assemblies (which, in turn, can reference more assemblies), thereby establishing an application composed of many components rather than existing as one large, monolithic program. This is an important feature that modern programming frameworks take for granted, because it significantly improves maintainability and allows a single component to be shared across multiple programs.

In addition to the manifest, an assembly contains the CIL code within one or more modules. Generally, the assembly and the manifest are combined into a single file, as was the case with HelloWorld.exe in Chapter 1. However, it is possible to place modules into their own separate files and then use an assembly linker (al.exe) to create an assembly file that includes a manifest referencing each module.⁴ This approach not only provides another means of breaking a program into components but also enables the development of one assembly using multiple source languages.

4. This is partly because one of the primary CLI IDEs, Visual Studio .NET, lacks functionality for working with assemblies composed of multiple modules. Current implementations of Visual Studio .NET do not have integrated tools for building multimodule assemblies, and when they use such assemblies, IntelliSense does not fully function.

Casually, the terms module and assembly are somewhat interchangeable. However, the term assembly is predominant when talking about CLI-compatible programs or libraries. Figure 24.2 depicts the various component terms.

Note that both assemblies and modules can also reference files such as resource files that have been localized to a particular language. Although it is rare, two different assemblies can reference the same module or file.

Even though an assembly can include multiple modules and files, the entire group of files has only one version number, which is placed in the assembly manifest. Therefore, the smallest versionable component within an application is the assembly, even if that assembly is composed of multiple files. If you change any of the referenced files—even to release a patch—without updating the assembly manifest, you will violate the integrity of the manifest and the entire assembly itself. As a result, assemblies form the logical construct of a component or unit of deployment.

Even though an assembly (the logical construct) could consist of multiple modules, most assemblies contain only one. Furthermore, Microsoft now provides an ILMerge.exe utility for combining multiple modules and their manifests into a single file assembly.

Because the manifest includes a reference to all the files an assembly depends on, it is possible to use the manifest to determine an assembly’s dependencies. Furthermore, at execution time, the runtime needs to examine only the manifest to determine which files it requires. Only tool vendors distributing libraries shared by multiple applications (e.g., Microsoft) need to register those files at deployment time. This makes deployment significantly easier. Often, deployment of a CLI-based application is referred to as xcopy deployment, after the Windows xcopy command that simply copies files to a selected destination.

Common Intermediate Language

In keeping with the Common Language Infrastructure name, another important feature of the CIL and the CLI is to support the interaction of multiple languages within the same application (instead of portability of source code across multiple operating systems). As a result, the CIL is the intermediate language not only for C#, but also for many other languages, including Visual Basic .NET, the Java-like language J#, some incantations of Smalltalk, C++, and a host of others (more than 20 at the time of this writing, including versions of COBOL and FORTRAN). Languages that compile to the CIL are termed source languages, and each has a custom compiler that converts the source language to the CIL. Once compiled to the CIL, the source language is insignificant. This powerful feature enables the development of libraries by different development groups across multiple organizations, without concern for the language choice of a particular group. Thus, the CIL enables programming language interoperability as well as operating system portability.

Common Type System

Regardless of the programming language, the resultant program operates internally on data types; therefore, the CLI includes the Common Type System (CTS). The CTS defines how types are structured and laid out in memory, as well as the concepts and behaviors that surround types. It includes type manipulation directives alongside the information about the data stored within the type. The CTS standard applies to how types appear and behave at the external boundary of a language because the purpose of the CTS is to achieve interoperability between languages. It is the responsibility of the runtime at execution time to enforce the contracts established by the CTS.

Within the CTS, types are classified into two categories:

• Values are bit patterns used to represent basic types, such as integers and characters, as well as more complex data in the form of structures. Each value type corresponds to a separate type designation not stored within the bits themselves. The separate type designation refers to the type definition that provides the meaning of each bit within the value and the operations that the value supports.

• Objects contain within them the object’s type designation. (This helps in enabling type checking.) Objects have identity that makes each instance unique. Furthermore, objects have slots that can store other types (either values or object references). Unlike with values, changing the contents of a slot does not change the identity of the object.

These two categories of types translate directly to C# syntax that provides a means of declaring each type.

Common Language Specification

Since the language integration advantages provided by the CTS generally outweigh the costs of implementing it, the majority of source languages support the CTS. However, there is also a subset of CTS language conformance called the Common Language Specification (CLS), whose focus is on library implementations. The CLS is intended for library developers and provides them with standards for writing libraries that are accessible from the majority of source languages, regardless of whether the source languages using the library are CTS-compliant. It is called the Common Language Specification because it is intended to also encourage CLI languages to provide a means of creating interoperable libraries, or libraries that are accessible from other languages.

For example, although it is perfectly reasonable for a language to provide support for an unsigned integer, such a type is not included as part of the CLS. Therefore, developers implementing a class library should not externally expose unsigned integers because doing so would cause the library to be less accessible from CLS-compliant source languages that do not support unsigned integers. Ideally, then, any libraries that are to be accessible from multiple languages should conform to the CLS. Note that the CLS is not concerned with types that are not exposed externally to the assembly.

Also note that it is possible to have the compiler issue a warning when you create an API that is not CLS-compliant. To accomplish this, you use the assembly attribute System.CLSCompliant and specify a value of true for the parameter.

Metadata

In addition to execution instructions, CIL code includes metadata about the types and files included in a program. The metadata includes the following items:

• A description of each type within a program or class library

• The manifest information containing data about the program itself, along with the libraries it depends on

• Custom attributes embedded in the code, providing additional information about the constructs that the attributes decorate

The metadata is not a cursory, nonessential add-on to the CIL, but rather represents a core component of the CLI implementation. It provides the representation and the behavior information about a type and includes location information about which assembly contains a particular type definition. It plays a key role in saving data from the compiler and making it accessible at execution time to debuggers and the runtime. This data not only is available in the CIL code but also is accessible during machine code execution so that the runtime can continue to make any necessary type checks.

Metadata provides a mechanism for the runtime to handle a mixture of native and managed code execution. Also, it increases code and execution robustness because it smooths the migration from one library version to the next, replacing compile-time–defined binding with a load-time implementation.

All header information about a library and its dependencies is found in a portion of the metadata known as the manifest. As a result, the manifest portion of the metadata enables developers to determine a module’s dependencies, including information about particular versions of the dependencies and signatures indicating who created the module. At execution time, the runtime uses the manifest to determine which dependent libraries to load, whether the libraries or the main program has been tampered with, and whether assemblies are missing.

The metadata also contains custom attributes that may decorate the code. Attributes provide additional metadata about CIL instructions that are accessible via the program at execution time.

Metadata is available at execution time by a mechanism known as reflection. With reflection, it is possible to look up a type or its member at execution time and then invoke that member or determine whether a construct is decorated with a particular attribute. This provides for late binding, in which the system determines which code to execute at execution time rather than at compile time. Reflection can even be used for generating documentation by iterating through metadata and copying it into a help document of some kind (see Chapter 18).

.NET Native and Ahead of Time Compilation

The .NET Native feature (supported by .NET Core and recent .NET Framework implementations) creates a platform-specific executable. This is referred to as ahead of time (AOT) compilation.

.NET Native allows programmers to continue to code in C# while achieving native code performance and faster startup times by eliminating the need to JIT compile code. When .NET Native compiles an application, the .NET FCL is statically linked to the application; .NET Framework runtime components optimized for static pre-compilation are included as well. These specially built components are optimized for .NET Native and provide improved performance over the standard .NET runtime. The compilation step does not change your application in any way. You are free to use all the constructs and APIs of .NET, as well as depend on managed memory and memory cleanup, since .NET Native will include all components of the .NET Framework in your executable.

Summary

This chapter described many new terms and acronyms that are important for understanding the context under which C# programs run. The preponderance of three-letter acronyms can be confusing, so Table 24.2 summarizes the terms and acronyms that are part of the CLI.