1 Introduction to Packer

A machine image is a static unit that contains a preconfigured operating system and installed software. You can use images to clone or create new hosts. Images help speed up the process of building and deploying new infrastructure. Images come in many formats, specific to various platforms and deployment environments.

In one of my first jobs I built servers. It wasn’t done using an API, the latest cool Cloud service, or even via a configuration management tool. Instead we wheeled a trolley, stacked with four PCs, a CD burner, and an assortment of cables and CDs, around the data center. We plugged directly into freshly racked servers and loaded previously built images, usually called “golden images,” onto the new servers manually and slowly.

For many years, this approach remained the state of the art for building servers. The process was slow and painful. It also created hard-to-manage servers. Every image was a snapshot, a point-in-time collection of packages and configuration. Each server was generally fixed at that point in time. This meant your images quickly became out of date and potentially vulnerable to security issues in older packages. Mass updates—for example, package updates—were possible but tricky and, because the “golden image” model predated configuration management tools like Puppet, mass configuration changes were complex and error prone.

Additionally it was rare that a single image was sufficient for every server. Some servers had shared base configuration but they were rarely entirely fit for purpose. This problem was again because this model predated configuration management. In these ancient times, additional configuration of hosts was rarely more sophisticated than scripts run at startup.

To partially address this issue, images began to multiply. Version 1 of the golden image became Version 2. Version 2 spawned into Version 2 Variant 1, customized for the New Jersey data center, and Version 2 Variant 2, for the data center in Palo Alto. Other variants emerged, perhaps customized for a specific server role. This would be exacerbated by image requirements for different platforms or operating system versions or hardware variants. The result is what can be best described as “image sprawl”: your environment ending up containing thousands of images.

Not only would this mean every server was subtly different, but a server’s ancestry was often a convoluted mess of inherited images. This combined with the existing entropy and change in an environment resulted in tens, hundreds, or even thousands of servers that were allegedly built the same way but were, in reality, unique.

This model only started to change as configuration management tools began to evolve. Tools like Puppet now allowed you to build fleets of servers and customize or maintain them via configuration management manifests. Images often remained—to deal with base hardware differences or to provide a starting step or bootstrap for a server—but much of the work they had performed was trimmed away.

The pressure off, image management became a less important concern. The focus on post-install configuration meant that the state of the art for image production and management went largely unchanged for a number of years.

This status quo remained unchanged until new trends in server and compute resource usage started to emerge. This was marked by increased use of virtual machines, both locally in data centers and in cloud services like Amazon Web Services. All of these new technologies use images. Some images underpin base virtual machines like the Open Virtualization Format (OVF); others, like the Amazon Web Services’ Amazon Machine Image (AMI) format, underpin Amazon’s EC2 compute instances.

In a further evolution, engineer Mitchell Hashimoto released a tool called Vagrant. Vagrant was designed to help manage local development environments and relied on shared images called “boxes” from which virtual machines are cloned and launched. Vagrant also supports provisioners. Provisioners launch post-build activities, such as configuration management tools.

Lastly, container technologies, especially Docker and Kubernetes, have the concept of images as one of their base building blocks. Indeed, container technologies regularly bake applications and application code into images, not just infrastructure.

All of these changes resulted in an increased use of images. They also exposed the fact that many of the same issues—entropy of images, the complexity of image maintenance and management, and keeping images up to date—continued to exist in some form, even when images were used in conjunction with configuration management tools building and customizing on top of base images.

In response to this increased use of images, a number of new image management and pipeline approaches and tools have emerged. One of the first was Veewee. Veewee was written by Patrick Debois and designed to help build Vagrant boxes and virtual machine images. Veewee relies on the concept of building images based on customizable template files. It combines these templates with additional scripts to build and configure images. For many, building base images and launching them with Vagrant, then using Vagrant’s provisioning integration to run a configuration management tool to complete the build process, was an attractive approach.

Building on this concept, Mitchell Hashimoto and the team at Hashicorp released a new tool called Packer.