Continuous Package Integration
Having read the previous chapters of this book, you have now reached the point where you can start supporting processes in your development environment by using NuGet. You know how to create and publish packages and you have also learned how to set up your own NuGet feeds. Great! Now, let’s fit the pieces together and streamline our development processes, because from a business point of view, that’s one of the main reasons to introduce a package manager into a development organization.
You may already be using some common concepts and techniques to mitigate the risks of development, such as continuous integration and unit testing, perhaps even integration testing and automated deployments, or maybe a fully streamlined continuous delivery process. If you don’t do any of these already, now would be a very good time to start.
This chapter will demonstrate how you can use NuGet in your development environment to support these processes. We’ll focus primarily on supporting good practices for the application life cycle. We’ll discuss whether or not NuGet packages belong in source control and look at the options available to facilitate both scenarios. Ensuring that packages flow from development to release without modification or rebuild is a typical scenario that deserves some attention as well; we’ll call it package promotion later in this chapter.
When it comes to source control, it is very interesting to see how development teams approach external software dependencies, or software dependencies in general. Many of us use a dedicated folder to store them in source control, often called ReferencedAssemblies, References, Lib, or something similar. This is a well-known strategy for keeping solution-level software dependencies close to the source code.
Many developers have been applying this good practice for years, yet there is no single convention on how the folder should be named or structured or where it should be located. This kind of information is typically defined in the corporation’s naming conventions and guidelines documentation. It is strange to see a multitude of different guidelines, often with variations in underlying reasons, solving a common problem we all share. NuGet offers an opportunity to have a standardized location and name for those references by using the default $(SolutionDir)packages folder. Even though this default can be overridden, it is a step toward a common direction.
In addition to this rather cosmetic annoyance (something many developers experienced when trying to figure out conventions while joining a new project or team), you might also experience issues with your version control system. Team Foundation Server (TFS) is a common victim of rants against its behavior for merging binary files. Ever had the task of upgrading a dependency and trying to check it into TFS source control, only to be prompted by a dialog box stating that TFS did not see any changes and leaving you with no other option but to undo your pending changes for that binary file?
Although blaming a tool is always easier than questioning the process, you have to realize that a version control system is not designed to version binary files but rather to track textual files such as source code.
The bigger issue that comes with storing all those software dependencies in source control is the amount of disk space required to store all those files, which are often duplicated in multiple project repositories. What’s the point in storing NHibernate version 2.2.0 in source control, next to all of your 37 solutions? All of them are the same. What kind of history will you keep in your version control system for these files? You won’t change them yourself, so all you keep track of is that, at a certain point in time, you added, removed, or changed the version of a dependency. The amount of useful information, in bytes, is almost nil compared to the amount of storage required for the dependency itself. Agreed, disk space is inexpensive nowadays, but we all want our sources to be safe from failure, right? Redundant, fail-safe storage that is backed up at regular intervals contributes to requiring even more disk space.
Also, do the binaries for these external dependencies belong in your source code repository? Is the actual binary a requirement for your software? Or would an identifier for it, such as a name and version number, be enough to rebuild a specific version of your software?
In NuGet terms, all you need is a package ID and a package version. The information stored in the packages.config file in your project contains everything you need to manage your dependencies. You store it in a version control system (VCS), after all.
Does this mean that you can choose to no longer commit packages into your VCS? Yes! But before you do, you need to put in place a good process supporting this scenario. It’s not a lot of work; it’s not overcomplicated, and it’s quite fast to set up.
Keep in mind that we don’t want to break any other established good practices by banning NuGet packages from source control. If you want to keep storing those packages next to your sources, that’s fine. It’s your choice. We don’t want to force you in any way. But if you do want to get out of the comfort zone and gain insights into an alternative approach, you might want to give it a try and see for yourself. In fact, we want to make sure that every single build you do, based on a given revision or changeset of your sources, even without the required NuGet packages in source control, always produces the same output. The end result of building with or without NuGet packages in source control must be the same. The no-commit strategy outlined in the following sections explains how to achieve this while not storing those dependencies in source control.
As stated before, NuGet has a default setting as to where packages are being located relative to the Visual Studio solution. Depending on the VCS tool you use, there are various ways to keep NuGet packages out of your source control repository. We’ll stick to the default package installation directory of $(SolutionDir)packages. Conceptually, all you need to do is to ignore the entire directory, except for the repositories.config file.
The next sections will describe ignoring the $(SolutionDir)packages directory when using some popular VCS tools such as Mercurial, Git, Subversion, and TFS.
When using the Mercurial (HG) source control system, you can add this directory to your ignore file. The Mercurial system uses a file called .hgignore in the root directory of a repository to control its behavior when it searches for files it is not currently tracking. More information about its syntax can be found at the following URL: www.selenic.com/mercurial/hgignore.5.html.
Tip If you’re not familiar with the .hgignore file, you can easily install a base ignore file for each of your projects by using NuGet (what else?). Simply install the HgIgnore NuGet package by running the Install-Package HgIgnore command in the NuGet Package Manager Console.
A good baseline for an .hgignore file when doing .NET development looks as follows:
syntax: glob
TestResults/*
*[Bb]in/
*[Dd]ebug/
*[Oo]bj/
*[Rr]elease/
*_[Rr]e[Ss]harper.*/
*.docstates
*.user
*.suo
*.xap
This is also the default one that is contained in version 1.4 of the HgIgnore NuGet package. However, this one still keeps track of the NuGet packages installed into the $(SolutionDir)packages folder, which is the default location defined by NuGet.
To make sure you track only the $(SolutionDir)packages epositories.config file and no NuGet packages, you should add the following to your .hgignore file:
syntax: regex
^packages/(?!repositories.config)
Another way is to use the HgIgnore NuGet package we created for you, available on the Pro NuGet MyGet feed of this book. Install it by executing the following command in the NuGet Package Manager Console:
Install-Package HgIgnoreNuGet –Source https://www.myget.org/F/pronuget
Git has a very similar concept for ignoring files and folders in source control, by using a .gitignore file. You can define ignore settings globally, on a per repository basis, or even on the folder level. Detailed information about how Git handles the .gitignore file can be found at the following URL: http://help.github.com/ignore-files/.
To prevent NuGet packages from being tracked by the Git VCS, you can put an .ignore file in the $(SolutionDir)packages folder with the following contents:
# ignore everything in this folder
*
# do not ignore the .gitignore file
!.gitignore
# track the repositories.config file as well
!repositories.config
If you prefer having just one global .gitignore file in the root of the repository, adding two lines to that file will do:
# ignore nuget packages
packages/
# but not repositories.config
!packages/repositories.config
Subversion (SVN) also supports ignoring files. However, it does not use any ignore file to achieve this. Instead, SVN works with ignore lists and SVN properties, which makes ignoring the $(SolutionDir)packages folder, except the repositories.config file, a bit more cumbersome. We will use TortoiseSVN in our example, because it is the most popular SVN utility out there.
TortoiseSVN has Windows Explorer shell integration. When you open a local source folder with an SVN repository, you’ll have SVN functionality readily available in the right-click contextual menu in Windows Explorer. When using SVN, you have to reverse your logic a bit. Instead of ignoring, we’ll be adding only the files we want to the repository. To track the $(SolutionDir)packages epositories.config file, you simply right-click the $(SolutionDir)packages folder in Windows Explorer and choose the Add option in the contextual menu that appears, as shown in Figure 6-1.
Figure 6-1. Adding an item to the SVN repository by using TortoiseSVN
After doing so, you’ll be prompted with a dialog box, where you’ll have more options to specify what exactly you want to track in your SVN repository, as shown in Figure 6-2.
Figure 6-2. Selecting what to add to the SVN repository using TortoiseSVN
By selecting only the $(SolutionDir)packages folder and the repositories.config file, we are not keeping track of everything else inside the package installation directory. All other contents will be treated as unversioned.
TFS does not deal with ignore files as such. You have to think in terms of workspaces and cloaking, which might make you feel a bit like Darth Vader designing the Death Star. It is, however, quite easy to achieve the same result once you know where to look.
The no-commit strategy tells you not to commit any NuGet packages into your VCS, but as most people using TFS have mapped the entire branch to their workspace, this causes issues. Whenever we install a package through Visual Studio into a mapped workspace folder, Visual Studio tells TFS to add these files to source control. We want to keep track of the repositories.config file but not of the actual packages, which are, by default, being installed into the same $(SolutionDir)packages folder. Respecting NuGet’s convention-over-configuration approach, we want to keep these defaults.
The workaround is not very straightforward when looking at the UI, yet it’s not hard to do from the workspace mapping dialog box. If you thought you could map only folders, think again! You can map single files as well! Knowing this is half of the work; the other part is just setting the correct workspace mapping, as shown in Figure 6-3.
Figure 6-3. Cloaking the packages folder (except repositories.config) in a TFS workspace mapping
The no-commit strategy is a specific approach to dealing—actually, to not dealing—with NuGet packages in source control. You might be wondering what the trade-offs are for this approach, or which benefits you get vs. what kind of drawbacks to expect. As with everything, criticism is justified. Every approach has its benefits and its drawbacks. It’s only a matter of picking the approach that works best for you in your situation.
Let’s discuss the aspect of storage. One could say that disk space is inexpensive nowadays. And as a matter of fact, that is very true! If you have abundant storage, and you have no issue in duplicating those references over and over again, one could very easily agree with you. But again: you do want to make sure that source code is on fail-safe storage, preferably on redundant disks and being backed up at regular intervals, making the storage requirements a lot bigger all of a sudden.
Even when space is abundant, you might consider a no-commit strategy in any case. There is, however, one thing that is not cheap, and most likely will never be. It’s usually left out of the picture in these kinds of discussions—time. Time is expensive. In fact, we often find ourselves lacking time. Duplicating your binaries in source control not only requires disk space but relies on processing power as well. What happens if you branch your project? Are those binary files copied into the new branch? Or is your VCS tracking only the delta of those files? Do you know the answer to this question with absolute certainty, without looking it up on the Internet? As usual, it depends. It depends on the source control system you are using. Distributed version control systems (DVCSs), such as Git or Mercurial, tend to behave very differently than centralized version control systems, such as TFS. The way your VCS will deal with those binaries during a branch (or merge) operation heavily depends on its (in)capability of tracking history on those files. For instance, TFS cannot track any differences for a .dll file; hence it cannot calculate the delta between two changes. Because of that, it will do a full copy of this file when it is branched. On the other hand, it will most likely not be able to merge changes for those files as well. You’ll have to choose between the server version and your local version. This, again, takes time—time from the CPU tracking your sources, as well as time from the developers dealing with this kind of problem. Did we mention the time needed to perform backups of your source repository? Extrapolate the impact of time on your operational costs, and you might have found another compelling argument to convince your manager to consider another approach. Also, bear in mind that you don’t want to track history on the actual content of those binary files, especially for external dependencies. What you really want to track is its metadata: the package ID and version information.
What would you say if centralizing your dependencies could save you time? Some teams already took a similar approach in the pre-NuGet era. They usually dedicated some network share to store all binary references and pointed all their projects to reference assemblies directly from that location. And the funny thing is that most of them are complaining about the time it takes to build. A significant delay is experienced when building your projects referencing binaries located somewhere on the network. This is obviously caused by the amount of network traffic required to fetch those binaries during the build process.
NuGet, however, has built-in support for caching of NuGet packages. By default, every new package you fetch from a NuGet feed is cached on the local machine. This is true both for your development machine and for any build server that fetches a NuGet package. This is a significant benefit over the custom solution of referencing binaries directly from a network share. Instead of downloading the binaries upon every single compilation of your project, you now need to download them only once. Every subsequent build of your project referencing those exact same packages will not experience the delay caused by network traffic and will fetch those packages from the local cache.
In addition to the caching optimization, NuGet packages are also compressed. As we mentioned earlier, a NuGet package is an archive file, which you can uncompress using any popular archive tool out there, such as our favorite, 7-Zip (www.7-zip.org). This is another optimization to increase the speed at which packages are downloaded, or in other words, decrease the amount of time required to fetch those NuGet packages.
So far, the trade-offs give the no-commit strategy the benefit of the doubt—a centralized location for your references, less disk space required to store them, and less time spent on tracking changes, dealing with conflicts, or across the wire generating unnecessary amounts of network traffic.
There is, however, one subtle dependency for this approach, which, at the same time, is its major drawback: a network connection to the package source is required. When the package source cannot be queried for the necessary packages required to build your project, compilation will fail—obviously because of the missing references. Note that after checking the package hash with the package source, the local cache will be used to install packages if they are present there. If we analyze the impact of this drawback, it is rather limited.
A build server needs a connection to your source repository as well, so on that level, your build server and source control repository share the same risk. If the package repository is on a different server than your sources, you just added another point of failure, because you now have two different connections that can fail. You have various options, though, for hosting your package repository: on the same server as your sources, on the build server (if you only have one), or even in the cloud (MyGet).
Your development machines share the exact same drawback as the build server. If a developer cannot build the solution because his laptop cannot connect to the NuGet repository, he has an issue. Developers don’t like issues. Developers like good (or better) practices. The developer’s workstation does not need to be connected with the NuGet package repository all the time, though. As with the build server, the connection is required only if the developer is performing operations against the NuGet repository. The actions a developer typically performs against a NuGet repository are limited as well:
- Updating a NuGet dependency to a newer version
- Installing a new NuGet dependency
- Fetching a NuGet package that is not yet available in the local cache
- Pushing or publishing a package to the NuGet repository (Shouldn’t this be done as part of the release process, through an automated build, for instance?)
If you analyze the use cases in the preceding list, you should notice that none are actions a developer performs every day. Preferably, these actions are well planned and at least have some kind of standardized process. Updating a NuGet dependency or installing a new one should not be an ad hoc decision. Neither is releasing a package to the repository, ready for consumption by everyone pointing to it. This is not something you will do when working on the train home.
The use case of fetching a NuGet package that is not yet available on the local cache of the developer’s workstation, however, is one to keep in mind, because it might happen more often than you think. Imagine that someone else in your team upgraded a NuGet dependency or installed a new package, and you just got the latest version out of your VCS, eager to continue coding on that same train home. If you did not build your solution and fetched those new NuGet packages before leaving the office, you’d be in for a frustrating ride.
As such, we want to introduce a good practice: whenever you fetch the latest sources from your VCS, rebuild your solution, and make sure you have all packages required for compilation to succeed. There’s an even better approach: communicate changes in project dependencies to the team! Make it part of the process of updating or installing new dependencies. If you don’t have a process, then this is the minimum. Communication is key!
NuGet in its most pure form, the command line and its packages, do not rely on Visual Studio or PowerShell. If you check the installed NuGet packages into your VCS, not a single developer is required to have any NuGet tooling installed on his system. That is good! We don’t want to break this by requiring all developers to have this tooling installed on their systems. Even though it is very easy to get up and running fast, we consider it clean. When applying a no-commit strategy, we need to respect that. You’ll read later on in this chapter how to do so.
In a nutshell, all we want to have in our VCS is a trace of which dependencies we have and which versions we depend on. In NuGet terms, we’re talking about package IDs and package versions. This happens to be exactly what is being stored into the packages.config files that NuGet produces when managing NuGet packages in a project or solution. Next to that, there’s also the repositories.config file, by default located under the $(SolutionDir)packages folder. The repositories.config file, as its name suggests, keeps track of the different NuGet repositories that your solution projects rely on and is thus pointing to all packages.config files for a given solution. This is the kind of meta-information we want to keep track of.
The information contained within these files is everything you need to fetch the required NuGet packages before building your project. This form of tracking your dependencies is very elegant. Every VCS is able to deal with XML files, which are just simple textual files, so we benefit from the comparison and tracking capabilities. The few merge conflicts you might encounter on these files will be very easy to resolve, as you’ll be able to actually compare the contents of these files.
The main reason that we do want to track this information lays in one of the most important principles behind the continuous integration process: a given changeset in your VCS must always produce the same build output. This effectively means there is no room for tampering with any file you get from source control before unleashing the compiler on it to produce its build output.
Another key practice is to use only trusted sources when continuously integrating your product on the build server. That’s the only way to ensure that no one else has modified any input to the build server. The need for this practice is very obvious for your source code files, which are being tracked in your VCS: you can easily view the history of your sources and track changes made to your build server. This is less obvious but equally as important for NuGet packages that come from outside your VCS. Make sure you can trust the NuGet package source you are using for your builds. This could result in limiting write permissions on the package source you are using—for instance, a network share, an internal NuGet server, or a secured MyGet feed. This could also mean you want to track who modified what in your NuGet repository, through RSS activity feeds, for instance, or custom logging you implemented in your NuGet server. In addition, this could even mean that every project has its own NuGet repository, which again is very easy to achieve using MyGet.
Tip If you experience issues connecting to an internal VCS and an external NuGet repository, or vice versa, whether on the build server or local machine, you might need to add both of them to the Trusted Sites list in Internet Options. This is definitely the case when your development workstation or build server is behind a corporate proxy.
Another great opportunity resides in the fact that software dependencies are now easily discoverable in these XML configuration files. Any decent software factory wants to know the answers to the following questions at any time:
- Who is using version X of component Y?
- Which versions of component Z are in use (and which not)?
- What dependencies does application A have and to which versions?
Answering these questions typically requires someone to maintain a spreadsheet with dependency matrices. What if you could just analyze these files with a tool and automate the generation of a package dependency matrix?
Tip In an attempt to automate dependency matrices, Xavier has built a proof-of-concept NuGet command-line extension that allows you to analyze a TFS or filesystem repository for NuGet package dependencies. It is open source and available on GitHub (patches and feedback are much appreciated): https://github.com/xavierdecoster/NuGet.Analyze.
Try it out by using the following commands in the NuGet Package Manager Console:
Install-Package NuGet.InstallCommandLineExtension
Install-CommandLineExtension NuGet.Analyze
To perform continuous integration on your sources that depend on NuGet packages that are not in your VCS, we need to restore those packages before building. This is what we call continuous package integration.
An example process flow for continuous package integration can be found in Figure 6-4.
Figure 6-4. A basic continuous package integration process flowchart
Up to version 1.5 of NuGet, this could be achieved using David Fowler’s NuGetPowerTools package (Install-Package NuGetPowerTools), which you can find at www.nuget.org. Because of the success of this approach, the NuGet team decided to integrate this functionality in NuGet starting from version 1.6, making it unnecessary for you to reinstall the NuGetPowerTools package for every single solution. And with version 2.7, a new approach to package restore was introduced. In there, it’s enabled by default and completely transparent.
We will not cover the NuGetPowerTools package, but since not everyone may already be using NuGet 2.7 we will cover the pre-2.7 era and the post-2.7 era.
Package Restore Prior to NuGet 2.7
The Enable Package Restore feature built into NuGet 1.6 and above allows you to very easily set up the prebuild part of the workflow. To do so, right-click the solution node in Visual Studio’s Solution Explorer, and click the Enable NuGet Package Restore option. If you do, and you still don’t see this menu item appear, you either already enabled this option, or you have a folder named .nuget in your solution directory. Figure 6-5 illustrates this option.
Figure 6-5. The Enable NuGetPackage Restore option in Visual Studio’s Solution Explorer
Visual Studio will prompt with a dialog box explaining what is about to happen (“A .nuget folder will be added to the root of the solution that contains files that enable package restore. Are you sure?”). Simply confirming with Yes sets up NuGet package restore. You can now delete all subfolders of your package installation directory, by default $(SolutionDir)packages, except for the repositories.config file, and your solution should still compile properly. During compilation, you should see NuGet installation traces in the Visual Studio output window, and you should see the required NuGet packages reappear in the package installation directory as well.
Although it is very easy to set this up, we encourage you to read further and get to know the details of how this works. If you ever run into issues (due to a merge conflict, for instance) and notice strange behaviors, it helps you a lot if you know what is going on.
After enabling NuGet package restore, you’ll notice a .nuget folder in your solution directory containing three files: the NuGet command line (nuget.exe), a nuget.config file, and an MSBuild file called nuget.targets.
Caution Do not forget to check the .nuget folder into your VCS, because your solution now depends on it to be able to compile successfully.
If you know that your Visual Studio project files are MSBuild files as well, you conceptually already know what’s going on. What happens is that, when enabling NuGet package restore, NuGet will iterate over all your solution’s projects that contain NuGet package dependencies and add an import statement in the project file. The import statement will import the nuget.targets MSBuild file. It will also add a RestorePackages property to your project and set it to true. Listing 6-1 shows you the most meaningful parts of the nuget.targets task, allowing you to understand what’s going on.
Listing 6-1. Partial NuGet.targets MSBuild Task Used for Package Restore
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns=" http://schemas.microsoft.com/developer/msbuild/2003 ">
<PropertyGroup>
<SolutionDir Condition="$(SolutionDir) == '' Or $(SolutionDir) == '*Undefined*'">
$(MSBuildProjectDirectory)..
</SolutionDir>
<!-- Enable the restore command to run before builds -->
<RestorePackages Condition=" '$(RestorePackages)' == '' ">false</RestorePackages>
<!-- Property that enables building a package from a project -->
<BuildPackage Condition=" '$(BuildPackage)' == '' ">false</BuildPackage>
<!-- Determines if package restore consent is required to restore packages -->
<RequireRestoreConsent
Condition=" '$(RequireRestoreConsent)' != 'false' ">true</RequireRestoreConsent>
<!-- Download NuGet.exe if it does not already exist -->
<DownloadNuGetExe Condition=" '$(DownloadNuGetExe)' == '' ">false</DownloadNuGetExe>
</PropertyGroup>
<ItemGroup Condition=" '$(PackageSources)' == '' ">
<!-- Package sources used to restore packages. By default, registered sources under %APPDATA%NuGetNuGet.Config will be used -->
<!-- The official NuGet package source ( https://nuget.org/api/v2/ ) will be excluded if package sources are specified and it does not appear in the list -->
<!--
<PackageSource Include=" https://nuget.org/api/v2/ " />
<PackageSource Include=" https://my-nuget-source/nuget/ " />
-->
</ItemGroup>
...
</Project>
It helps if you are a bit familiar with MSBuild, but it’s not really needed. Just know that an additional RestorePackages target, triggering a command-line nuget install packages.config, is going to be executed as part of every build, for each Visual Studio project that has the property RestorePackages set to true. You can easily access this file from within Visual Studio in the .nuget solution folder that got added to your solution, as shown in Figure 6-6.
Figure 6-6. The .nuget solution folder added by the Enable NuGet Package Restore feature
Tweaking the NuGet.targets File
The NuGet.targets file contains all the logic for making package restore possible. It contains a series of MSBuild properties that can be changed, as well as the actual package restore target that calls into nuget.exe. Table 6-1 lists the MSBuild properties you may want to change.
Table 6-1. NuGet.targets MSBuild Properties
| Property | Default value | Description |
|---|---|---|
| SolutionDir | $(MSBuildProjectDirectory).. | The solution directory for the solution in which NuGet package restore will run. |
| RestorePackages | false | Is package restore enabled? Each project file will typically override this variable and decide whether package restore is enabled. |
| BuildPackage | false | Should a NuGet package be created for every project? This can be overridden in the project file. |
| RequireRestoreConsent | true | Should consent be given for package restore? By default, NuGet package restore will prompt the user for license acceptance, essentially blocking the restore process for input. If you want to run NuGet package restore on a build server, change the value of the RequireRestoreConsent element to false. This will ensure no license prompts block the NuGet executable. |
| DownloadNuGetExe | false | Should a copy of nuget.exe be downloaded from the Internet? |
Next to these MSBuild properties, you can also specify the feed(s) from which packages should be downloaded by modifying the PackageSources setting. If you want, for instance, to fetch NuGet packages from the ProNuGet feed on MyGet, you could use the following:
<PackageSources> https://www.myget.org/F/pronuget</PackageSources >
By default, the package sources defined in the nuget.config file will be used. However, we prefer explicitly indicating the package source in this file, even when you are using only the NuGet.org feed, so you can track changes to it. It is also the only way to ensure that no external changes can break your build and to ensure that you always get the same output for a given changeset. If any external influence changes the nuget.config file on the machine that compiles your project, NuGet might be redirected to a different package source than the one you intended.
It is important to point out that the NuGet package restore feature will install the exact same package version as defined in the project’s packages.config file: it will not perform any upgrades for the exact same reasons.
Package Restore and Build Targets Originating from a NuGet Package
Package restore may sound like the Holy Grail to some, but it also comes with a caveat: you have to know what you’re doing—but then again, isn’t that the case with everything?
Imagine you are making use of a package that contains custom build targets, a feature we described in Chapter 3. An example of such a package is the Microsoft.Bcl.Async package, which adds support for async/await in .NET 4.0, Silverlight, and Windows Phone 7.5 projects. Somewhere in its dependency chain (in the Microsoft.Bcl.Build package, to be precise), it adds an MSBuild target file to your project to be able to do all compiler magic that is required for making async/await work.
Note If you haven’t heard of async/await or are interested in what happens behind the curtains, Dino Esposito has a great introductory article on this at www.simple-talk.com/dotnet/.net-framework/syntactic-sugar-and-the-async-pill/.
The Microsoft.Bcl.Build.targets file injected into your project is located under the packages folder. Now guess what happens if we enable package restore and don’t add the contents of the packages folder to our source control system. Depending on how the package was created, one of two things will happen:
- If the package was not created using the uild folder convention introduced with NuGet 2.5 (see Chapter 3) and hasn’t been properly added to the project file, the project will no longer load in Visual Studio.
- If the package follows the uild folder convention, NuGet will restore the .targets file and nothing bad will happen.
Unfortunately, a lot of packages are still not making use of this new package convention, resulting in not being able to load a project in Visual Studio or failing builds on a build server. There is a simple solution to this problem, though, which consists of making a small modification to the project file referencing the MSBuild .targets file by checking whether the .targets file exists.
You can open the project file that isn’t loading by using a text editor like Notepad or Notepad++ and find the <Import /> element in which the .targets file is referenced. For the Microsoft.Bcl.Build package (version 1.0.7), this element looks like the following:
<Import Project="..packagesMicrosoft.Bcl.Build.1.0.7 oolsMicrosoft.Bcl.Build.targets" />
We can modify the statement by adding a condition to it, checking whether the file exists on disk. The added attribute is shown in bold:
<Import Project="..packagesMicrosoft.Bcl.Build.1.0.7 oolsMicrosoft.Bcl.Build.targets" Condition="Exists('$(SolutionDir)packagesMicrosoft.Bcl.Build.1.0.7 oolsMicrosoft.Bcl.Build.targets')" />
If you save the project file and load it again in Visual Studio, at least the project will load. However, depending on what features in the imported .targets file are being used, it may not be sufficient to get package restore working. The only option there is to run package restore before the build kicks off, by running nuget.exe install packages.config. On a build server you can do this by running the following batch file to build the project:
REM Package restore
.nuget uget.exe install MySolutionpackages.config -OutputDirectory %cd%packages -NonInteractive
REM Build
%WINDIR%Microsoft.NETFrameworkv4.0.30319msbuild MySolution.sln /p:Configuration="Release" /m /v:M /fl /flp:LogFile=msbuild.log;Verbosity=Normal /nr:false
The easiest solution to the problem described in this section is probably just making sure the imported .targets file is stored in your version control system. With NuGet shipping new releases, most packages adding MSBuild targets will hopefully start respecting the uild folder convention, making this fix obsolete by the next edition of this book. And when using NuGet 2.7’s package restore feature, this fix is already obsolete.
Package Restore with NuGet 2.7 and Beyond
The NuGet team started doing things differently with NuGet 2.7. Before that version, package restore was essentially a hack of the Visual Studio project file, sneaking in a custom build target that ran nuget.exe install. As we’ve seen in the previous section, running package restore during the build instead of prior to it sometimes caused trouble.
With the release of NuGet 2.7, package restore is no longer the responsibility of the Visual Studio project file but of the build system. Or in other words: Visual Studio or your build server are now responsible for running package restore prior to starting the build.
The NuGet Visual Studio extension does this by default, but this behavior can be changed through the options, under Tools ä Options ä Package Manager ä General. Figure 6-7 shows this dialog box and relevant settings.
Figure 6-7. Package restore options in Visual Studio
As you can see, the new package restore functionality is much easier to use and comes with less hassle: no longer do you have to commit a nuget.config file and a copy of the NuGet command-line tool to your source control system.
When enabled, Visual Studio will run package restore before starting the actual build of the project files. Essentially, it calls into the NuGet command line behind the scenes, running nuget.exe restore <your solution file>.sln. As such, this is the key to package restore with NuGet 2.7 and beyond: calling into the nuget.exe restore command. If you need package restore on your build server, call into this command prior to the actual build. If you are writing your own build system, call into this command prior to the actual build. There are some additional options that can be specified on the call to nuget.exe restore. These are described in Appendix A.
NuGet packages, just like any other release vehicle (binaries, installers, and so forth), need to undergo rigorous testing and quality assurance before being released to the masses. One cannot just build a complex NuGet package, complete with custom PowerShell hook points, push it to the repository, and assume everything will work as expected, every single time, over and over again. If you’re that good, you should be able to convince your managers to code directly onto the production servers.
We like Eric S. Raymond’s approach of release early, release often. This is exactly what you should do with your NuGet packages as well. You release not only the package contents, but also the package itself. It is your release vehicle. That is why your continuous integration builds should be producing NuGet packages as well if you intend to ship your product as a NuGet package (or multiple NuGet packages).
Note The concept of release early, release often (or RERO, in short) actually has another part of the sentence: “and listen to your customers.” Besides ensuring quality, anticipating your customer’s needs and gathering feedback are among the most important requirements for your product to become a success. Continuously delivering high-quality releases enables you to do so. In addition, it’s the fastest way of delivering value to your clients.
Phil Haack has a great blog post on this topic: http://haacked.com/archive/2011/04/20/release-early-and-often.aspx .
As your package goes through various stages of testing, you want to ensure nothing gets changed in those packages. No one is permitted to modify the package file or its contents. To facilitate the different teams that are involved in this process, one could choose to set up various NuGet feeds. An example setup is as follows:
- CI feed: Every CI build pushes its output onto this feed, ready for consumption by development teams who want to play around with it, but more important, for QA teams to test them.
- QA feed: All packages from the CI feed that have been validated and approved by QA could then be pushed onto the QA feed. Whenever a release needs to get out, you can pick a version and push it to the Release feed.
- Release feed: This feed will be used for general consumption of your officially released packages.
With this setup, you have a very short track of releasing a package, while ensuring optimal quality and respecting the normal release processes. We like to call this flow NuGet package promotion. How you should move these packages from repository to repository is explained in the section “Phasing Out Packages” later in this chapter.
Next to package restore, NuGet v1.6 also introduced the concept of prerelease packages. A prerelease NuGet package is considered unstable, and thus not ready yet for general consumption. A prerelease package is often indicated by the commonly used terms alpha, beta, RC, and so on. The concept of prereleases does not fit well with the concept of release fast, release often that we discussed in the previous section. If we cannot modify the package contents (for example, the version number), and we don’t know up front whether our package will make it through the QA gate, how do we know when to tag our package with a certain release number? This is contradictory and counterintuitive. Then why does NuGet support prerelease tags at all?
Let’s look at things from another perspective. We all know requirements change. Quite often, this happens very fast, and it occasionally impacts planning as well. Well, it nearly always impacts planning. It could be in the time dimension, resulting in shifting release dates. It could be in the scope dimension, resulting in adjusted functionality for a milestone. It could also have another impact on planning in the broader meaning of the word, any type of planning, including capacity or resource planning.
If planning is affected in the time dimension, you can choose to pick the latest QA-approved package and give it a prerelease tag before pushing it onto the Release feed. Probably more commonly, the opposite can happen as well, usually when changes have been made in scope: pick a prerelease package from QA, drop all version information after the dash (thus removing prerelease information), and push it out as an official release.
As explained in Chapter 3, NuGet also respects the SemVer notation and meaning of a prerelease package. Prerelease information is appended to the patch number, prefixed with a dash, as shown here:
major.minor.patch-prereleasetag (e.g. 1.0.0-RC1)
In our opinion, even prereleases are official releases. They indicate only a notion of potential instability or potential feature incompleteness. Prereleases should be considered milestones and be planned accordingly. A prerelease has a specific purpose—you want to get early feedback from your customers. This means that, ideally, your prerelease tags in the version information of your package should already be present in the CI build output.
In an enterprise environment, it is quite common to stop supporting an older, or maybe obsolete, framework. The same thing can happen to one or more of your NuGet package dependencies. If you want to discontinue a NuGet package, you either want to make sure that no one is still depending on it or that at least you can provide them with a working upgrade path.
A good example is the NuGet.org feed: in the past, package owners could delete a package from the gallery. A package owner had no idea who depended on that package and had no way of learning that either. A package owner could guess purely on the download count and, if by keeping track of this number over time, spot a trend in the usage of the package. However, a declining trend still does not justify the rude deletion of the package from a publicly available package source. Some package owners deleted their packages, essentially breaking other people’s development and builds. This is the reason why the NuGet.org gallery now supports only “unlisting” packages instead, keeping them available for download but just no longer showing them in search results. This decision made it possible for package owners to still retract their package from search results but not break anyone’s process.
What should enterprises do with their desire to block certain packages from being installed by their development teams? Deleting any already listed packages from your package source can have a significant impact. As with anything, communication is key. Communicating that support for a certain package (version) will be discontinued is a better practice. At least, your developers are aware that they should mitigate the risk associated with this pending change.
If you are using multiple package sources, you could add one for archiving those old or obsolete packages. If you delete a package without keeping a copy just in case, you’re in for trouble. Archiving a package, after communicating its retirement, doesn’t sound like a bad thing to us.
You could easily perform these actions by just moving the .nupkg file from one location to another, assuming you have enough permissions on both package sources. Moving those packages from filesystem path to filesystem path is straightforward. Moving a package from a feed to a filesystem path implies that you can download the package, store it on the archiving location, and delete it from the originating feed. This requires many more security permissions to be set properly. No matter how you move a package from one feed to another, you’ll need all these permissions. This is a clear indication of how big an impact this action can have on your package consumers. Be cautious.
You could reuse Rob Reynold’s command-line extension for NuGet, the NuGet.copy.extension package. You can install it by executing the following commands in the NuGet Package Manager Console:
Install-Package NuGet.InstallCommandLineExtension
Install-CommandLineExtension NuGet.copy.extension
After installing this extension, your NuGet command-line instances should have a new command available: the copy command. This command allows you to copy a package from one feed to another, by specifying the package ID (with optional version number), a source, a destination, and optionally an API key for the destination.
An example usage of the copy command follows:
nuget copy jQuery –Version 1.9.1 –Source https://www.myget.org/F/pronuget –Destination
https://www.myget.org/F/pronugetarchive
When you are using MyGet feeds, you might as well benefit from the built-in package retirement feature. MyGet gives you the option to set, on a per package basis, whether the package is listed or not, without actually deleting it from the feed. By unlisting a package, you are preventing it from being consumed. Any newer versions of the package that are listed will still be available—older ones as well, but you should probably ask yourself why you didn’t unlist those too. This effectively means you are no longer supporting a specific package (version) while maintaining an upgrade path to a newer version for those who still rely on it. Remember, you should communicate a package retirement so people can take appropriate actions, such as mirroring the package you are about to retire or even check it into source control as a temporary measure. This is a nice, soft phase-out of a package. In fact, you give your consumers a polite and gentle push toward upgrading this dependency.
Caution Continue respecting core principles of software craftsmanship: package owners can decide when to stop support for a certain package, but it is still the consumer’s decision when and if to upgrade to a newer version!
Listing or unlisting a package on a MyGet feed is the same as toggling support on or off for that package. When navigating to the package details page on MyGet.org, you’ll see a button next to each package version in the Package History listing, as shown in Figure 6-8.
Figure 6-8. Listing and unlisting NuGet packages on a MyGet feed
If you toggle the state of a package version from listed to unlisted, you’ll be prompted by a dialog box warning you about the impact of your action, which you’ll have to confirm, as shown in Figure 6-9.
Figure 6-9. A dialog box will warn you and ask for confirmation when unlisting a package on MyGet
In addition, because of the soft implementation of this functionality, the operation of unlisting a package can be undone, as opposed to deleting the package from the feed.
Summary
In this chapter, we focused on supporting the development environment and processes, as well as the application and package life cycle. We’ve explained how NuGet can live in perfect symbiosis with established techniques such as continuous integration.
We have provided you with guidance on how you can set up a no-commit strategy using various version control systems and considered a different approach of dealing with software dependencies outside your VCS. We discussed both benefits and drawbacks of the no-commit strategy and highlighted some of the trade-offs being made.
Using a NuGet package as a release vehicle of our product, we discussed how packages should respect a correct flow and how versioning and package promotion can help you in doing so. We’ve touched on the enterprise scenario of gently phasing out old or obsolete packages as well.
The next chapter will change perspectives from continuously integrating to continuously delivering and discuss optimizations in the field of automated delivery, using NuGet.