Usually, compilers behave as black boxes or like a function in the source code, where the code to be compiled is the argument, there's something going on in the middle, and an output is generated at the other end. The process entails an inner, deep understanding of the code they are dealing with, but such information is not available to developers. Besides, this information is dismissed after the translated output is produced.

The mission of Roslyn is to open the black box and allow developers to not only know what's going on behind the scenes, but—ultimately—also have the capability to create their own tools and code checkers and extend the traditional possibilities created by old compilers.

The official documentation for Roslyn (https://github.com/dotnet/roslyn), explains the main changes of this approach by comparing the classical compiler pipeline with the set of services proposed by Roslyn:

As the figure shows, every part of the pipeline has been replaced with APIs that allow you to write code that can be parsed, create Syntax Tree API, and generate a whole symbol map out of it, performing the required Binding and Flow Analysis APIs in order to finally use the Emit API to generate the resulting binaries.

The way Roslyn handles these phases is by creating object models for each of them. A deep study on the capabilities and opportunities offered by this set of services and tools is beyond the scope of this book, but I would like to present an introductory view of these possibilities, along with some demo code so that it's possible to start building you own utensils: projects that read code and help identify potential issues and how to fix them.

There are some requirements that apply before you start using Roslyn from Visual Studio. The first one is to have the Git extension installed: you can find it—as with many others—in the Extensions and Updates tool in the Tools menu.

After installation, create a new project in Visual Studio, select the C# Language, and under the Extensibility item, choose Download the .NET Compiler Platform SDK, as shown in the following screenshot:

An index.html web page will appear, containing a button linked to downloads for syntax tree visualizers, templates for analyzers, and so on. Note that if you have more than one edition of Visual Studio installed, you will be notified by the .vsix installer about which products you want the extensions to be installed in.

Several options appear as available in different contexts now. On the one hand, if you go to the Tools/Options menu and check the Text Editors item, you can find new options to control the way this language is managed within Visual Studio Editors on the C# side: options to format code for Intellisense, and so on.

On the other hand, after reloading Visual Studio, if you go back to Extension and Updates, you will find that new types of projects are available now, including Stand-Alone Code Analysis Tool, Analyzer With Code Fix (NuGet + VSIX), Code Refactoring (VSIX), and VSIX Project, this last one being specific to installations of plugins and the like. You should get an offer like the one shown in the following screenshot:

Let's start with a simple class and see what we can do with the options. So, I create a new project (a Console one is just fine) and get rid of the using declarations included by default.

Even with the default initial code we have, Roslyn will read and convert it into a Syntax Tree representation, where everything (every word, whitespace, curly brace, and so on) has a place in the tree and can be managed accordingly. This tree can be examined using the new window available in View | Other Windows | Syntax Visualizer installed by the previous process.

As soon as we click on the source code (that is, in the middle of the class word), the window will show the result of the code analysis (we show the legend as well):

You'll notice that the tree starts with something called CompilationUnit, with the main NamespaceDeclaration node hanging from it. Therefore, every single element in the source code is now recognizable and manageable.

If we want to see this tree in a more visual manner, we can right-click on the CompilationUnit contextual menu and select the View Directed Syntax Graph option, which will show a .dgml file in the editor with a colored tree in which every color in the legend represents one element in the code.

When passing the mouse over one element, its properties are shown in a tooltip (also, right-clicking on a single node shows a contextual menu of possible options):

The blue nodes characterize the high-level nodes of the C# grammar that can be divided into smaller units. The green ones are called syntax tokens and are, somehow, like the atoms or basic units of the syntax tree (they cannot be divided into anything smaller).

The rest of the nodes (white and gray nodes) are the so-called trivia nodes, and they're not relevant to compilation as they are the parts of the source text considered largely insignificant for normal understanding of the code, such as whitespace, comments, and preprocessor directives, according to the official documentation.

Besides, there is another very useful online tool (open source) called Source Visualizer, which is available at http://source.roslyn.io/ and shows how Roslyn is coded, along with its source code.

You're allowed to navigate through the whole tree of elements found in the Roslyn project and check them out, reviewing how they are coded to serve as an inspiration for your own code.

For example, if we click on the left tree in the search for the CSharp compiler, we can see how it is coded and all the details linked to it, as the following screenshot shows:

Along the course of this book, and probably long before, you may have noticed the large amount of options available within the source code editors in order to facilitate usual operations, notify errors before compilations, and suggest changes among other things (remember, for example, when talking about the implementation of the IDispose interface, how the IDE suggested several possible implementations for us).

From Visual Studio 2015 onwards, these features are just some of the many tools powered by Roslyn. One of the most popular among them is the set of services linked to Code Analyzers.

An entire open source sample for you to check: ScriptCS

There's an open source project that can give you an idea about some of these possibilities. It's called ScriptCS. Remember, we mentioned that with Roslyn, you can build a tool similar to the REPL (read-evaluate-print-loop) available for Node.js, Ruby, and Python, for example. I mean a tool that can examine and execute C# code.

To test it, just go to the ScriptCS website (http://scriptcs.net/) and download the project. It's a Visual Studio solution made up of several projects that shed some light about the possibilities this technology offers.

Once compiled, if you launch the program, you'll see a console application, which suggests that you write some code to analyze and execute. The tool will use the compiler, and it works in a manner very similar to the Console tool in browsers.

The aspect will be like what is shown in the following screenshot. Note that I write three separate sentences, and only after writing the one that produces an output, we get the results in the console:

Of course, Roslyn services are creating a class behind the scenes for us and are inserting that code within, later calling the compiler, executing the code, and redirecting the output to the Console window, where we see the results.

It becomes useful when we just want to check out a simple piece of code without building a whole project.

A basic project using Microsoft.CodeAnalysis

Let's start working with these tools, creating a simple Console application and installing Microsoft.CodeAnalysis tools directly from NuGet Package Manager Console.

We can type Install-Package Microsoft.CodeAnalysis, and we'll see the installation process in which all the required dependencies are downloaded, with the last message shown saying something like Successfully installed 'Microsoft.CodeAnalysis 1.3.2' to [TheNameOfYourProject].

In the main method, we are going to load a C# file in order to analyze its contents. With this purpose, we have created a Person.cs file with the following contents:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace ConsoleRoselyn1
{
  class Person
  {
    public int id { get; set; }
    public string Name { get; set; }
    public int Age { get; set; }
    internal Person Create()
    {
      Person p = new Person();
      p.id = 1;
      p.Name = "Chris Talline";
      p.Age = 33;
      return p;
    }
    internal string PersonData()
    {
      var p = Create();
      return p.id.ToString() + "-" + p.Name + "-" + 
        p.Age.ToString();
    }
  }
}

Later on, we're defining a new entry point, InitialParser.cs, which is going to take care of the analysis. We'll establish this class as the entry point of the application, and in its main method, we start by reading the file to be checked, using the same class as earlier (CSharpSyntaxTree)—only, this time, we load the file contents early in order to pass them to the ParseText static method of the class:

// Fist, we localize and load the file to check out
string filename = @"[Path-to-your-Project]Person.cs";
string content = File.ReadAllText(filename);
// Now we have to analyze the contents
// So, we use the same class as before. Notice 
// it returns a SyntaxTree object.
SyntaxTree tree = CSharpSyntaxTree.ParseText(content);

Observe that ParseText returns a SyntaxTree object. This is fundamental for analysis since it allows you to iterate over the whole tree in order to inspect how the Tree Object Model was implemented when it was applied to our Person class.

If you want to have a clear view of why certain objects are selected to recover the code's properties, remember that the Syntax Tree Viewer that we discussed earlier achieves many of the actions that we are going to perform here, and it offers the corresponding name of the element as we move from one point in the code to another.

For instance, if you click on the code right inside the class keyword, the Syntax Tree Visualizer will move exactly to that point in the tree, indicating the name associated with the Object Model, as the next screenshot shows:

Now, we have a very nice tool to suggest which classes and classes' members we should identify in the API in order to obtain references to the elements that compose the Syntax Tree.

If we want to obtain the name of the first class defined in the code (there's only one, but the syntax tree will show as many as there were), first, we need to access the root of the tree. We do that by calling GetRoot() in the tree object previously obtained.

Once we have the root element, a look at the methods used throws some light on the possibilities we have. Here are some of these methods, just to name a few:

• We can go down or up, looking for descendants in search of ancestors since we have access to the whole list of nodes
• We can find a given node or check the contents of any node in search for something special
• We can read a node's text
• We can even insert or remove any of them (refer to the following screenshot):

Given that all collections provided by the APIs are generic collections, we can ask for nodes of a concrete type using the OfType<element> syntax. That's what we do next in order to get the ClassDeclarationSyntax object of our Person class, so we print it to the console as follows:

ClassDeclarationSyntax personClass = root.DescendantNodes().OfType<ClassDeclarationSyntax>().First();
Console.WriteLine("Class names");
Console.WriteLine("-----------");
Console.WriteLine(personClass.Identifier);

We can go on and obtain the method's names in the class using the objects already declared. So, in this case, we'll ask for all the MethodDeclarationSyntax objects available after the DescendantNodes() call and go through them, printing their names:

Console.WriteLine("
Method names");
Console.WriteLine("------------");
personClass.DescendantNodes().OfType<MethodDeclarationSyntax>().ToList().ForEach(method => Console.WriteLine(method.Identifier));

So, we can go for the properties, knowing that the syntax tree categorizes them as PropertyDeclarationSyntax objects:

// And the properties
Console.WriteLine("
Properties");
Console.WriteLine("----------");
personClass.DescendantNodes()
.OfType<PropertyDeclarationSyntax>().ToList()
.ForEach(property => Console.WriteLine(property.Identifier));

The previous code generates the following output:

This is one of the recommended procedures to iterate over the syntax tree and recover information about its members, although in this case, we're just reading data and presenting the results.

Based on the previous ideas and APIs, let's look at how to program those diagnosing and refactoring features that Visual Studio offers. That's the main reason for Extensibility.

Just remember something about the building and parsing behavior of Visual Studio. Many of these features are disabled by default. The whole set of analysis' capabilities is found—for any project—in the Project/Properties/Code Analysis tab and is presented with two main options:

That said, we're going to create one of these projects that appeared after the installation of the SDK that we did earlier.

What we'll do is select the type of project named Analyzer with code Fix (VSIX) and look at how it is programmed and what the principal units of code are. Then, we'll cover debugging since it works in a peculiar way with respect to any other debugging scenario.

After creating the new project, you'll notice the presence of three individual projects in the solution: the analyzer itself, another one for testing purposes, and finally, the one with the .vsix extension, which serves as the deploying mechanism.

Let's focus on the first one. To honor its name, there are two classes implied: one for analysis (DiganosticAnalyzer.cs) and another in charge of code fixing (CodeFixProvider.cs). It's important to recognize these roles and keep the code like this, even when we want to extend the default functionality.

It doesn't matter that the project's purpose is a bit simple: it searches for a class definition that contains a lowercase letter and marks it as a target for CodeFixProvider to advise about this situation.

To perform this first task of finding the code, the Analyzer2Analyzer class, which inherits from DiagnosticAnalyzer performs the following actions (we explain them one by one since it's not obvious at first):

As we can see, although there are not many lines, it's not a simple code. That's why we need to understand how the internals of Roslyn work in order to follow the right actions involved in the context to check for a certain issue. The complete code is as follows:

using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.Diagnostics;
using System.Collections.Immutable;
using System.Linq;

namespace Analyzer2
{
  [DiagnosticAnalyzer(LanguageNames.CSharp)]
  public class Analyzer2Analyzer : DiagnosticAnalyzer
  {
    public const string DiagnosticId = "Analyzer2";

    // You can change these strings in the Resources.resx file. If you do not want your analyzer to be localize-able, you can use regular strings for Title and MessageFormat.
    // See https://github.com/dotnet/roslyn/blob/master/docs/analyzers/Localizing%20Analyzers.md for more on localization
    private static readonly LocalizableString Title = new LocalizableResourceString(nameof(Resources.AnalyzerTitle), Resources.ResourceManager, typeof(Resources));
    private static readonly LocalizableString MessageFormat = new LocalizableResourceString(nameof(Resources.AnalyzerMessageFormat), Resources.ResourceManager, typeof(Resources));
    private static readonly LocalizableString Description = new LocalizableResourceString(nameof(Resources.AnalyzerDescription), Resources.ResourceManager, typeof(Resources));
    private const string Category = "Naming";

    private static DiagnosticDescriptor Rule = new DiagnosticDescriptor(DiagnosticId, Title, MessageFormat, Category, DiagnosticSeverity.Warning, isEnabledByDefault: true, description: Description);

    public override ImmutableArray<DiagnosticDescriptor> SupportedDiagnostics { get { return ImmutableArray.Create(Rule); } }

    public override void Initialize(AnalysisContext context)
    {
      // TODO: Consider registering other actions that act on syntax instead of or in addition to symbols
      // See https://github.com/dotnet/roslyn/blob/master/docs/analyzers/Analyzer%20Actions%20Semantics.md for more information
      context.RegisterSymbolAction(AnalyzeSymbol, SymbolKind.NamedType);
    }

    private static void AnalyzeSymbol(SymbolAnalysisContext context)
    {
      // TODO: Replace the following code with your own analysis, generating Diagnostic objects for any issues you find
      var namedTypeSymbol = (INamedTypeSymbol)context.Symbol;

      // Find just those named type symbols with names containing lowercase letters.
      if (namedTypeSymbol.Name.ToCharArray().Any(char.IsLower))
      {
        // For all such symbols, produce a diagnostic.
        var diagnostic = Diagnostic.Create(Rule, namedTypeSymbol.Locations[0], namedTypeSymbol.Name);

        context.ReportDiagnostic(diagnostic);
      }
    }
  }
}

Although the counterpart (CodeFixer) has some more lines of code, you will be able to read the rest of the code—and understand how it operates—by taking a look at Analyzer2CodeFixProvider included in the CodeFixProvider.cs file.

The two important methods here are the override to RegisterCodeFixesAsync, which receives CodeFixContext (required to launch the fixing action) and the fixing action represented in the demo by the MakeUppercaseAsync method.

When this method is called, it returns a Task<Solution> object and receives all the required information to perform the task, plus a CancellationToken object to allow the user to ignore the fix suggestion offered in the contextual dialog box. Of course, it's responsible for changing the code if the user accepts the modification.

Debugging and testing the demo

To test these demos, a new instance of Visual Studio is launched, which will have the Analyzer registered and active when loaded. For this case, I launched the project, and in the new instance of the IDE, I opened the previous project to understand how it recognizes names of identifiers with lowercase letters.

So, proceed in this manner, and open our previous Person.cs file (or any other similar file for that matter) to check this diagnosis in action. You will see a red squiggling underline on the declaration of the Person class.

When you place your cursor underneath the word Person, a tooltip will show up, advising you of a potential problem (in this case, there's no problem at all):

Up until this point, we were dealing with the first class analyzed (the Analyzer2Analyzer class). But now, we're offered a double option: the yellow bulb, with a contextual menu and the Show potential fixes link. Both take to the same window, showing the potential fixes in all places where this fix could be applied.

Also, note how these fixes are marked with color. In this case, the color is green, indicating that fixes will not provoke another diagnosis issue, but if it does, we will be notified accordingly:

We also have the option of Preview changes, which, in turn, presents another (scrollable) window in order to examine in detail what would happen to our code if we accept the suggestion (shown in the following screenshot):

To deploy the project, you can follow two different approaches: use the NuGet package generated (you will see it in the Bin/Debug folder after compilation as usual) or use the .vsix binaries generated by the compiler, which are also available in the same subdirectory, only in the Vsix project this time.

In the first case, you should follow the indications in the Readme.txt file (what follows is a citation of the previously mentioned file):

To try out the NuGet package:

1. Create a local NuGet feed by following the instructions here: http://docs.nuget.org/docs/creating-packages/hosting-your-own-nuget-feeds.
2. Copy the .nupkg file into that folder.
3. Open the target project in Visual Studio 2015.
4. Right-click on the project node in Solution Explorer and choose Manage NuGet Packages.
5. Select the NuGet feed you created on the left-hand side.
6. Choose your analyzer from the list and click on Install.

If you want to automatically deploy the .nupkg file to the local feed folder when you build this project, follow these steps:

1. Right-click on this project in Solution Explorer and choose Unload Project.
2. Right-click on this project and click on Edit.
3. Scroll down to the AfterBuild target.
4. In the Exec task, change the value inside Command after the –OutputDirectory path to point to your local NuGet feed folder.

The other choice is to launch the .vsix file (the extension will be recognized by the system). This will install the package in Visual Studio after asking for conformity, just like any other package that you might have previously installed.

Up to this point, we have introduced Roslyn and its capabilities. Now, we're going to visit the other big open source project, which is getting more hype due to its importance in many web projects, including Angular, Ionic, and many others: TypeScript.