Midnight Club Story

Of the score of games that I have worked on, only a few have shipped on time and within budget. Of these, Midnight Club was the most successful. Midnight Club was a PlayStation 2 (PS2) launch title where players illegally race in the large, open cities of London and New York. The project was a grueling 18-month struggle with a new platform and new publisher.

Our launch title was originally designed to include six cities. We considered two major risks to delivering a good game in time for the launch of the console. First was the uncertainty of developing for the PS2. The PS2 was powered by a custom CPU referred to as “the Emotion Engine,” marketed as a supercomputer on a chip. The hype of the machine’s power was so great that when it was revealed that Saddam Hussein had ordered 4,000 units, it was theorized he was going to use them to design nuclear weapons.¹

1. https://www.eurogamer.net/articles/article_29913

Working with the early development kits, we knew this wasn’t true. The only emotion we experienced was continual frustration with the weaknesses of the kits and the poor state of the support libraries and their documentation.

Second, we were uncertain about the effort to produce six large cities in the time we had before launch.

We focused pre-production on creating one city that would run well on the PS2. This effort took four times longer than we anticipated. Armed with this knowledge, we were forced to scale our plans down to three cities. Our publisher, Rockstar Games, took these cuts in stride. It knew how hard we were working.

During production, we slowly realized that we had a crucial decision to make. The time to create the necessary polish for deployable cities was taking even longer than we had planned. We had to choose between shipping three low-quality cities or two polished ones. After much teeth gnashing, Rockstar Games decided to ship with two cities.

In the end, it was the best decision and contributed to the game’s shipping as a PS2 launch title that sold millions of copies. A major reason for maintaining the schedule was the constant attention to the production costs of creating and polishing the cities. For many game projects—especially those with fixed ship dates—the production costs and schedule pressures create the greatest risks.

The Solutions in This Chapter

This chapter explores the challenges of managing a game project using Agile and Lean principles to

• Manage fixed ship dates

• Manage risk

• Manage project constraints such as cost, schedule, and scope

• Refine content production debt

• Choose when and how to employ Kanban practices for content production

• Optimize and improve the flow of content production work using Lean principles

Minimum Viable Game

Many projects outside the game industry are challenged with an obligation of delivering on certain expectations from the start. For example, a word processor would have to have undo, printing, font management, and so on. The feature and content set might require a year or more of effort before a minimum viable game (MVG) or the smallest possible set of features, content, and constraints required for a successful launch, are met.

Example MVG requirements are

• Gameplay content (levels, characters, and so on)

• Fixed ship dates such as holidays or movie co-release dates

• Minimum required feature sets—such as online multiplayer for a first-person shooter—that must be shipped with the game

These requirements demand budget and staff planning. For example, content production costs are refined over the course of pre-production releases to identify the number of content creators and the time needed to produce a minimum amount of gameplay time.

Minimum required feature sets necessitate similar long-term plans. For the first-person shooter example, the following is a minimum required feature set:

• Single-player gameplay

• AI

• Weapons

• Online multiplayer gameplay

These features create a debt of work, much like the content production debt, that takes multiple releases to deliver. Often, stakeholders demand that resource management plans be developed to support a predicted ship date for these features.

How does resource management planning work with Agile planning? A proven approach is to identify a range of resource needs to implement epic-level stories and continually refine both through Sprints and releases.

This is a balancing act of cost and value. Imagine you are buying yourself a new PC, but you have only $2,000 to spend, and not a cent more. You would start with a breakdown of the parts of the PC and the cost of each:

You might decide later to buy the latest graphics card that costs $100 more. This money has to come from another part of the computer. This is how long-term planning works with epics. Large epics are defined and thought about to produce a cost of time and resources. If any budget item is wrong, the time has to come from another bucket. That needs to be a deliberate and visible decision and needs to be publicly proclaimed.

For each item, the first estimate might be largely based on experience with previous games, which is not very accurate. For the first-person shooter example, a Product Owner identifies the large online multiplayer epics that a team of eight might complete in six to eight months. Some of these epics are not split until more is known. For example, cooperative online gameplay with AI may or may not be valuable or even possible within this resource plan until more is known, so the epic that might contain that feature is not split until more is known and a team is available to work on it.

This provides the flexibility necessary as more is discovered about the challenges, velocity, and opportunities for online as the game emerges. It does not ensure that every minimum required feature will be completed within budget and schedule—no planning process does that—but it enables decisions to be made earlier that help steer the project in the right direction.

Contracts

The typical game development contract has a series of defined milestones. Each milestone is associated with a specific date and features that need to be delivered on that date. If developers deliver those features on schedule, they are paid for the milestone. Milestone payments are the lifeblood for most independent developers; they will do almost anything to avoid missing a milestone. This includes avoiding change that would improve the game if it threatens the milestone with additional work. Who can blame them? Many developers who miss a milestone payment miss payroll; that is a very bad thing for them to do. The contract is an impediment to change.

On the other side, a publisher doesn’t have the full freedom to add features or change the milestone definition when it thinks the game would benefit from the change. The contract impedes working with the developer to fix the game.

Fixed-milestone deliverables have led to an adversarial relationship between developers and publishers. Both recognize the need for change to improve the game but lack the necessary level of trust to allow the change to occur.

Collaboration between developer and publisher should be valued more than a fixed contract. However, very few publishers allow a developer to work without a detailed contract. Outside our industry, many contracts in an Agile environment follow the time and materials form, which is where the client pays for the cost of the last iteration. This style of contract requires greater trust between both parties. The client has to trust that the developer is spending money wisely. The developer has to trust that the client won’t cancel the ongoing contract without good reason.

Although most Western publishers don’t support this model, many have adopted flexible milestone definitions that allow some level of collaboration with developers every few months. As the use of Agile spreads, we will see more collaborative business arrangements as trust is built through greater collaboration.

Hitting Fixed Ship Dates

There is an old project management saying: “Time, budget, scope—pick two,” which is fundamentally flawed. Projects always balance variables such as these and they are not independent of one another.

For example, it’s often assumed that we can throw more money at a game that’s late or in trouble and solve its problems. In practice this means adding developers, which usually puts the game in even greater peril because the new developers need to be brought up to speed by the developers currently working on the game, delaying it further.

Most plans for a new video game have a ship date in mind. However, very few actually fail if they miss that date. They’re mostly one of the main constraints that stakeholders put on teams to ensure they have a target to aim for. But some of the game teams I’ve worked with over the past decade do have critical ship dates. These are often sports titles that must be out for the start of a new season.

For these games, a disciplined Agile adoption has given them the best set of tools for hitting the fixed ship dates by

• Managing debt with a robust definition of done, automated testing, and solid technical practices

• Refining content production forecasts using tracer bullets while still in pre-production

• Managing scope in an ordered fashion to ensure delivering an MVG, while not risking the dates on non-essential scope

• Closely collaborating with the stakeholders on trade-offs

Managing Risk

Game development probably has to handle more risk than any other product outside of the defense industry. We have rapidly evolving, cutting-edge technology and fixed ship dates for sports seasons, movie releases, or seasonal markets. The market’s whims also change suddenly. As a result, many games don’t meet expectations or developers are asked to work miracles. Many of these problems could be avoided if projects embrace risk instead of avoiding it and accept that, inevitably, things go wrong from time to time.

The Need for Stages

For many games developed using Agile, a need still exists to separate some of the development activities. The three major reasons for this are

• Publishers require concepts: To gain publisher approval (which includes marketing and often franchise or license owner approval), developers need to create a detailed concept treatment at the start of a project. They are unable to stray too far from this vision throughout the project.

• Games need to deliver eight-plus hours of gameplay: Games typically deliver 8 to 12 hours of single-player gameplay. Games tell stories that need a great deal of production content to be created using mechanics discovered during pre-production.

• There is one launch: For many PC and console games, there is only one deployment date at the end of a 24+ month development cycle. Intensive hardware compliance testing by the first party platform owner is often delayed until the end.

The Development Stages

Agile game projects spread activities such as concept, design, coding, asset creation, optimizing, and debugging more evenly throughout their life. This doesn’t mean that the project is a homogenous string of Sprints.

Most game development projects have stages regardless of whether they are Agile or not. These stages change how teams develop the game:

• Concept: This stage occurs before pre-production. Concept development is almost purely iterative. Ideas are generated, possibly prototyped, and thrown away on a regular basis. This stage is usually timeboxed to deliver one or more concept development plans to a green-light approval process required by the publisher or a license holder.

• Pre-production: Teams explore what is fun and how they are going to build assets to support it during production. They also create levels and other assets that represent production quality. This stage is fully iterative and incremental. Teams iteratively discover what is fun and incrementally adapt development planning with this knowledge.

• Production: The team focuses on creating an 8- to 12-hour experience using the core mechanics and content production flow discovered during pre-production. This stage focuses on efficiency and incremental improvements. Teams iterate less on core mechanics discovered during pre-production because they are building a great deal of assets based on them. Changing those mechanics during production is usually very wasteful and expensive. For example, consider a team in production on a platformer genre game. Platformer games challenge the player to develop skills to navigate treacherous environments (such as Nintendo’s Mario series). The production team creates hundreds of assets that depend on character movement metrics such as “how high the character can jump” or “the minimum height that the player can crawl under.” If these metrics are changed in the midst of production, it wreaks havoc. For example, if a designer changes the jump height of the character, hundreds of ledges or barriers would have to be retrofitted. This creates a great deal of wasted effort during the most expensive phase of development. Discovering and locking down such metrics during pre-production is critical.

• Post-production: With the content brought to deployable quality, the team focuses on polishing the whole 8- to 12-hour game experience. This stage improves the game incrementally. Following this, the game is submitted to hardware testing. Although much of this testing is spread throughout the entire project, some of it cannot be. For example, Microsoft and Sony hardware testing is expensive and only occurs in the months prior to shipping the game.

Mixing the Stages

Stages aren’t isolated to distinct periods of time. For example, although a great deal of concept work is done up front, concept development needs to be refined as the game emerges over the entire project.

Figure 10.1 shows a typical distribution of efforts on an Agile game project. Note that although more design and concept is done up front and more tuning, debugging, and optimization are done at the end, many overlap with one another. For example, rather than an official “production start date,” teams see a gradual buildup of production activities and a drop-off of pre-production work in the middle of the project.

Managing Stages with Releases

Releases are a series of Sprints linked together to deliver major features and assets. Similarly, a game project is a series of releases that deliver a finished game to the consumer. Figure 10.2 shows a typical series of two- to three-month releases that make up a two-year project.

Each of these stages requires a different emphasis on the practices used. The transition between stages such as pre-production and production can be gradual with various asset types transitioning at different times.

The reasons for the change in practices are illustrated by an enhanced version of the Stacey diagram shown in Figure 10.3. As the game progresses from concept to post-production, the level of technical and requirements uncertainty drops. As the Stacey diagram indicates, the practices should reflect these levels of uncertainty.

The framework used is still Scrum, but teams adjust the practices for the current stage:

• Concept: Sprints are shorter, and most of the stories in the very small backlog are spikes. The main goal of the conceptual stage is to create knowledge for the team and stakeholders, not shippable value for the players. Release goals are concept treatments and perhaps a prototype to demonstrate to stakeholders.

• Pre-production: Scrum is used to discover the fun of the game and incrementally and iteratively build value and knowledge about production costs. Development is paced by Sprints and releases. Release goals are major features.

• Production: Teams produce assets that were identified in pre-production and incrementally improve the asset pipelines. Although Sprints and releases are still used, the pace of asset production becomes the metric for measuring velocity.

• Post-production: Teams focus on tuning, polishing, and bug fixing tasks they identify daily. Although Sprint and release reviews are held, the goals are driven more by the daily triage backlog (which includes bug fixes and polishing tasks) and upcoming key dates such as submission. Post-production starts on the alpha date and includes the beta and shipping dates.

Lean Production

Production is the most challenging and expensive stage. It represents a large debt of work that is dependent on pre-production decisions and timing.

For Scrum teams, the complex pipelines of asset creation don’t fit perfectly with the iterative flow of work in a Sprint. Because of this, many teams entering production abandon Scrum in favor of waterfall practices. The problem in doing this is that they abandon many of the Agile benefits.

This section addresses the issues with production for an Agile team. It introduces some Lean concepts that allow teams to continually improve their production rate and the quality of the assets they produce.

Production Debt

Have you ever seen a schedule for a game project that picks a day for production to begin? A date is chosen, a year or so in advance, that predicts when the team is to begin creating production assets (usually levels).

Where does this date come from? How do we know how much time is needed for pre-production and production? How do we know whether the time given is enough? How many times have you entered a 9-month production stage with 12 months of work? Many games enter production too early because the schedule says they need to do so. The problem is that the core mechanics and budgets that production depends on are still being iterated. This creates wasted work, because large numbers of assets must be reworked as this information emerges.

The work that needs to be done in production to create the 8 to 12 hours of content many games need is called production debt.

Measuring Production Debt in Pre-Production

One of the goals of pre-production releases should be to measure and refine knowledge about the size of the production debt. During the first few releases, that debt is uncertain. For example, a project early in development might estimate its production debt to be 1,000 people-months of work, plus or minus 20 percent. Changes to the feature set may impact those ranges as well. Toward the end of pre-production, it should be more accurate, such as 1,050 people-months, plus or minus 5 percent. Figure 10.4 shows how this range of estimates will change over time. Although this estimate will never be perfect, it is always better than the first guess made at the start of the project.

These estimates are refined with Sprint and release goals that call for assets with increasingly higher quality. By developing these assets, the team learns more about the actual cost required to create them. These estimates aren’t frozen in production either. During production, the team should be finding ways to improve how assets are created to further reduce costs.

Making Games is Like Making a Car

A lot of problems and solutions for game development have their parallels in automobile design and production. During the design of a car like the Prius, Toyota not only had to design a car that the green market would buy, but one they could mass-produce at a price point that the green market could afford. In other words, they had to design the factory at the same time they designed the car (Liker, 2004).

Why Measure Production Debt?

Measuring the cost of production during pre-production is important to help make decisions about the features that impact that cost. If Product Owners do not know the cost impact of such features, they are inclined to accept them at face value.

Imagine you are working on a first-person shooter game that has 12 months of production scheduled to build 10 levels. During development, the team implements technology that enables every part of the world to be destroyed or have holes blown in it. This is a great new addition, but it doubles the amount of work required to build a level.

If Product Owners know this, they can make an early decision. The following are some choices:

• Drop the feature because the project can’t afford it.

• Drop half the levels.

• Start production earlier.

• Extend production and the ship date.

• Scale up the production staff.

Some of these choices are better than others, but they are all better than the one taken if production debt isn’t measured: trying to stuff 24 months of production effort into 12 months of schedule.

The Challenge of Scrum in Production

Production is dominated by an assembly-line approach to asset creation characterized by sequential steps of specialized work. These asset production streams are easily stalled or starved of work at various points within by impediments.

When Scrum teams fit asset production streams into Sprints, they discover that some of the benefits they’ve enjoyed in pre-production, such as ad hoc iteration and transparency, are somewhat reduced.

Scrum Task Boards and Production Streams

At the start of a Sprint, a team forecasts the tasks it estimated it could complete by the end of the Sprint. Those tasks are placed on a task board that is reviewed every day. Many of the tasks are worked on out of sequence or in parallel. If someone is held up waiting for another task to complete, then he works on something else. This organic flow of task execution fosters communication among the team and prevents impediments from stopping it cold. Scrum task boards are great for supporting this.

However, for a long series of asset production steps that must be completed in order, the team loses much of this benefit. Tasks must flow in an ordered and steady pace to ensure that the specialists on a production team are not waiting for work. Scrum task boards don’t clearly show this flow.

For example, consider the asset production stream for a single character to be produced in a Sprint (see Figure 10.5). The team estimates it will complete this work before the end of the Sprint. As this stream shows, each task has to occur in order before it is handed off to the next one.

When this stream of work is placed on a task board, it looks like Figure 10.6.

This task board shows that the character model is being created first. One problem with displaying streams on a task board is that the flow rate of the stream is not sufficiently visualized. This lack of visualization fails to alert a team to problems.

For example, what happens if modeling takes longer than estimated? Figure 10.7 shows the likely result. The modeler, rigger, and animator are able to finish their work, but the last person in the chain, the audio designer, does not have enough time to do his work before the end of the Sprint.

The team needs to see the impact of delays within the stream while they are occurring at any time.

Keeping Everyone Busy

Another problem with fitting asset production streams into Sprints is keeping all the specialists busy. In the previous character production stream example, what is the audio designer doing before the animation is completed?

Scrum teams address this problem in a number of ways. One way is to share the composer with another team that needs audio. The other is to pool the audio designers into a team that has its own Backlog. Another is to batch up the audio work for a number of characters until the audio designer joins the team for a Sprint. None of these solutions is ideal because each increases the separation between an asset’s creation and its appearance in the game. For example, suppose modelers are pooled together where they create a dozen character models ahead of time. If a problem is revealed the first time one of the characters appears in the game, it might require major rework of all 12, which is a big waste of effort. Shortening the time between a model’s creation and its appearance on the screen creates more opportunities to “do it right” the first time.

Less Use of Iteration

Asset production pipelines are similar to assembly lines. In this sense, a Scrum team completely empties the assembly line by the end of every Sprint, which doesn’t make sense for assembly lines. They have to be continuously filled because each step in the line only works on what was passed from the previous step. An assembly-line run like a Sprint creates many gaps in the line that leave team members waiting.

This is the reason that some Scrum teams abandon Agile practices entirely when they enter production. However, production is never 100 percent efficient. The team cannot foresee every potential problem. It needs to seek improvements in pipelines right up until the game is shipped. For this reason, the Agile and Lean values and principles should be maintained in production. If production is driven by fixed schedules and deadlines, the best the team can hope for is to meet those deadlines, but unplanned problems will continue to appear and threaten them. Practices that are still Agile are needed—practices that anticipate change, that encourage continuous efficient flow, and that focus attention on continually improving how assets are produced.

This is where “Lean” thinking can help. As described in Chapter 2, Lean is a set of principles applied to production environments that have more certainty about what is being created but still want to introduce improvements continually. The remainder of this chapter describes how Lean practices such as Kanban can help a team remain Agile during production.

Lean Production with Kanban

As teams transition from the complex exploration of pre-production to the complicated flow of production, a Lean production method such as Kanban can be a better fit for their work.

Visualizing Flow with Kanban

A team employs a Kanban board to visualize an asset production stream and provide the transparency that a Scrum task board cannot.

We’ll use an example level production stream to demonstrate how Kanban and Lean thinking are applied to production. Figure 10.8 shows a simplified level production stream, from concept to tuning.

The first step is to represent the stream on a Kanban board, which uses columns to represent individual workflow steps and capacity. Figure 10.9 shows a simplified Kanban for the level production stream.

There are six columns, which represent the steps of this production stream, including the Product Backlog. The cards within the columns represent individual levels in production. As the work for each step of a level is completed, a card for the next level to work on is pulled from the column immediately to its left, if that level is ready.

Leveling Production Flow

Now that the team has a Kanban board representing the level production stream, the team members can start applying Lean tools to smooth out the fluctuations of production. This allows them to create production assets at a more constant and predictable rate.

There are two basic tools for leveling production flow. The first is establishing timeboxes for each step of the production stream. Following this, the flow is leveled by balancing the resources supporting every step.

Timeboxing

Every developer who uses Scrum recognizes that a Sprint is a two- to three-week timebox. The benefit is to create a measurable cadence of value being added to the game.

In Lean production, this is taken a step further. Each step of a production stream is often timeboxed. For example, audio design for a level might be timeboxed to 10 days. This is different from a similar Scrum task in pre-production that the audio designer independently estimates and commits to. This changes for production because the team has established, during pre-production, the ideal timebox for a level’s audio design. This timebox is based on the trade-off of time (cost) of creation and quality desired by the Product Owner.

Timeboxing does not mean content creators are forced to meet a set quality within a fixed amount of time. It’s an average. The quality is determined by what the content creator is able to provide within the time limit.

The key to timeboxing assets is balancing quality and cost, measured in time. If the timebox chosen is too short, then the quality of the asset will be too low. For example, if the timebox for a section of high-resolution level geometry is set to one day, an artist might only be able to create a section filled with untextured cubes! On the other hand, if the timebox selected were two months, the artist might deliver a section with too much detailed geometry. It would be absolutely beautiful, but that beauty would come at too great of a cost to the stakeholder compared to the value it provides to the player.

Balancing asset quality and cost is the responsibility of Product Owners. They have to judge what the player values. Figure 10.10 demonstrates a notional curve of the trade-off between cost and value to the player.

It’s not a straight line, because of diminishing returns; a player driving past a 10,000-polygon fire hydrant does not experience 50 times the value of a 200-polygon fire hydrant.

Note

As the Product Owner of a driving game, I asked the artists to create “95-mile-per-hour art.” This quality bar depends on what the player cares about when they play the game; the player doesn’t care about a high-resolution fire hydrant they pass at 95 miles per hour!

Figure 10.11 shows the area of the trade-off curve where the timebox selection is made. Pre-production refines the shape of this curve and the best range for a production timebox.

Timeboxes are not absolutely precise. Some levels take more time than others. The timebox chosen is an average of best- and worst-case times. The team uses small buffers, described next, to avoid underflow or overflow of work that this might cause.

Note

The timebox curve changes during production; it shrinks as it is refined and moves to the left as the team improves how it performs.

Leveling Work

Each step in the stream usually requires a different-length timebox. This causes gaps and pileups of work, which is to be avoided. For example, in Figure 10.12, if level design takes a week but the high-resolution geometry takes two, then the completed design work piles up. Conversely, if conceptualization requires two weeks for each level, the level designer eventually runs out of work and has to wait.

Workflow needs to be balanced so that everyone always has work to do and that large amounts of work in progress don’t pile up too much between any step. One way of doing this is to balance the amount of time each step takes, called its cycle time.

For example, if the team wants a 10-day cycle time for each step, it starts by examining the timeboxes for each person working on the stream (see Table 10.1).

Step	Timebox per Step
Concept	10 days
Level design	20 days
High-resolution art*	30 days
Audio design	10 days
Tuning pass	7 days

*High-resolution art is the creation of detailed geometry, textures, and lighting.

Concept art and audio design already have a cycle time of 10 days. However, the other steps have different cycle times. For example, tuning takes less than 10 days per level. This means the designer who tunes levels runs out of work from time to time and has to find other things to do. This could mean helping out another team, level design, or QA.

For steps that require more than 10 days per level, the team needs to add people or find improvements to the process. For example, because the high-resolution artists require 30 days per level, the team could dedicate three high-resolution artists to balance the flow. Artists can work together in a number of ways, but each has challenges:

• Have all high-resolution artists work on the same level: This may not be an option if the editing tools do not support simultaneous editing on the same level.

• Break up the high-resolution step into a specialized flow (for example, texture artist, prop artist, static geometry artist): It may be hard to balance these specialties.

• Have the high-resolution artists working on multiple levels in parallel: This solution might pose challenges with creating consistent quality and pace.

Each member of the team adds capacity, represented by a work in progress (WiP) limit for each column, as shown in Figure 10.13. This number defines the maximum number of cards (levels) that can reside in each column. When the number of cards in a column hits its WiP limit, no more cards can be pulled in.

Limiting Work in Progress

Limiting WiP is an essential part of improving flow and quality. Simply improving the productivity of each step is not enough.

A useful analogy is that of a pizza restaurant. Say you’re hungry, and you want to get a pizza. Your local shop makes great pizzas, and it takes them 20 minutes to bake one.

You enter the shop, but notice there is a long line. You’re not going to get your pizza in 20 minutes because of all those people waiting ahead of you.

That line of people is the WiP for the pizza shop. WiP adds delay. In game development, that delay results in

• Fewer iterations, therefore less quality and fewer opportunities to experiment with improving the flow of production

• More waste because if we find a problem when an asset finally gets into the game, there might be another dozen in the pipeline, due to a high WiP count, that need to be reworked

Teams will continually explore ways to reduce WiP limits at every stage of a pipeline.

Our Kanban board now looks like Figure 10.13.

The team has now balanced the level production stream and established the rate at which levels are completed, called the takt time (see the sidebar “Takt Time and Cycle Time”), which is also 10 days. They apply Lean tools to maintain and even reduce takt time so as to continually improve the cost and quality of the levels.

Takt Time and Cycle Time

Lean uses two measures of time for asset production streams. One is takt time, which is the rate of external demand for completed assets to be delivered. For video game production, this is determined by a schedule established in pre-production. One goal of Lean practices is to continually put pressure on reducing takt time—increasing the rate at which finished assets are delivered—by finding improvements in the pipeline.

Cycle time is used to measure the interval of time between the start and finish of a step or an entire stream.

Ideally, the cycle time of each step is less than or equal to the takt time. This is the goal of leveling flow. In the level creation example stream, the high-resolution art step required a 15-day cycle time. This could not be reduced to the takt time goal of five days by simply improving the pipeline. The solution was to create three high-resolution art stations working in parallel to meet the demand for one level of high-resolution art to be completed every five days.

Continual Improvement

One of the main advantages of applying Lean production is that teams maintain the drive to continually improve what they are making and how they are making it. This is a benefit over fixed production schedules, where teams concentrate on keeping pace with deadlines, and unanticipated problems add delay.

In production, teams apply the same planning paradigm of using velocity measurements to forecast the pace toward achieving a goal. This allows fixes and improvements to be quickly appraised.

This section uses the level production example to describe some of the ways Lean thinking enables teams to continually improve how they work together to increase the quality and production pace of assets.

Example Benefits of Kanban for Production

• “A lot more visibility and clarity on where things are and what needs to be worked on.”

• “Easy to quickly see the overall project schedule and what can and can’t get done. This helped many times for re-aligning resources to meet our goals.”

• “Planning is actually much easier and straightforward for a Kanban team working within the Scrum structure.”

• “Kanban got us to realize improvements to our process and pipeline. Specifically, making ‘blockout’ animations and ‘proxy’ models to get things in game as quickly as possible and keep everything moving.”

—Dante Falcone, Game Industry Producer and Engineer

Improving Cycle Time

The pace of iteration has a direct impact on quality. For example, if the wait time on changing a texture and seeing it in the game is reduced from 10 minutes to 10 seconds, an artist iterates on textures more, and the game looks better.

The same principle applies to asset production streams. We want to shorten a stream’s iteration time to allow improvements to be introduced as quickly as possible. These include pipeline, tool, workflow, and teamwork improvements. Shortening the iteration time of such changes enables more of them to be implemented and seen.

In Lean production, we focus on improving the iteration time, or cycle time, for the entire stream rather than the individual steps. The following factors influence a stream’s cycle time:

• Asset size: Large assets, such as full levels, have large cycle times. If a team completes portions of a large asset, it reduces asset cycle time.

• Batch size: This is the number of assets processed at one time at any individual step. The larger the batch size, the longer the cycle time. An example of this is completing a dozen character models before handing them all off to the rigger. This is one of the benefits of establishing smaller WiP limits.

• Waste: This is the effort spent on things that don’t add value to the final asset. For example, the time spent waiting for an approval is non-value-added time, or waste.

• Knowledge, skill, and empowerment: The greatest factor in determining cycle time is the knowledge and skill of everyone who adds value along the stream and their ability to influence change. For example, knowing when and where to reuse an asset rather than building one from scratch has a large impact on cycle time.

Smaller Assets

By breaking large assets into smaller ones, teams receive faster feedback. There are three types of feedback:

• Gameplay feedback: Very large assets, such as levels, can have month-long cycle times and, despite how detailed their design is, deliver uncertain gameplay. This long cycle time provides little opportunity for feedback to influence the asset. As a result, levels are shipped based on their initial design assumptions. By breaking levels into smaller areas, teams are given valuable feedback about the gameplay emerging and use this knowledge to influence subsequent portions of the level.

• Production feedback: Improvements to the production flow are applied more quickly and cheaply. For example, if the team discovers that a particular piece of static geometry interferes with character motion in the first section of the level, it fixes that section and changes its standards of work to apply the improvement on every subsequent section. This is a big time-saver.

• Velocity feedback: Gauging the effectiveness of individual practice changes or tool improvements when the cycle time is measured in months is very difficult. If the cycle time is reduced to weeks, then changes to the pace of work from improvements are more apparent.

One level production team I was working with did this by breaking up their levels into seven smaller “zones.” Zones were the sections of levels that were streamed in and out of memory as the player moved.

After they began building zones, their cycle times became one-seventh of what they were before. Zones were completed every few weeks, and each one added improvements in quality and production that fed into subsequent zones. This team eventually reduced its cycle time by more than 50 percent.

Smaller Batches

Traditional production pipelines focus on the efficient utilization of content creators, rather than the flow of asset production streams. The imbalance of disciplines often causes project managers to level the work being done so that nobody runs out of work to do. Because “discipline leveling” predictions are never too precise, large inventories of works in progress are built in between the steps.

For example, if a project has dozens of levels to ship, the team may start producing batches of concept art or level props well before the end of pre-production. To a certain degree, this is necessary, but it often goes too far. For example, a half-dozen levels of concept art created before gameplay is fully understood leads to waste if the concepts have to be redone or, worse, levels are produced using these obsolete concepts.

With Lean thinking, teams try to reduce or eliminate batches of work. Because of the uncertainties within the timeboxes for each step, teams often need to create small buffers of works in progress between the steps of an asset production stream. They choose the smallest possible buffer size, because buffers increase the cycle time. The buffers should prevent waiting but not excessive pileups of work between each stage of work.

Figure 10.14 shows how buffers are represented on a Kanban board. Buffers have WiP limits, like every other step, that signal when the buffer is overflowing or underflowing.

Reducing Waste

A good portion of the time spent in production is wasted on work or activities that don’t add value to the final product. Examples of this are the time waiting for exports, waiting for asset approvals, or syncing with the latest assets. Reducing these wastes greatly benefits productivity and cycle times.

Many of these wastes are identified and corrected by the team itself. The subtle pressures of takt time and timeboxing largely drive this. Timeboxes exert a pressure on the content creators to use their time wisely. As a result, it encourages them to seek ways to be more effective in how they work and point out the problems that were not so impactful when they had more time.

Limiting Creativity?

One concern about Lean is that it limits the creativity of artists in production. I have found that the opposite is the case. A quote from T.S. Eliot seems to apply: “When forced to work within a strict framework, the imagination is taxed to its utmost and will produce its richest ideas. Given total freedom the work is likely to sprawl.”

One production team I was working with had a 10-day takt time for level sections. This pace was challenging and exposed many problems with the flow of work. The biggest problem was with the concept art step. The team had only one concept artist available who was sitting with the other concept artists in another part of the studio. Creating a dozen drawings for each section often took more than 10 days. The team recognized this as a bottleneck.

In team discussions, it turned out that the level designers and high-resolution artists didn’t really need all the drawings created. Because the concept artist was separate from the team, much of the concept art was based on poor assumptions about the level and gameplay. For example, gameplay was linear, but much of the concept art represented open areas. The concept artist was surprised to hear that much of his work was useless. The solution the team created was to move the concept artist next to the level designer and high-resolution artists. This enabled them to discuss the layout of levels as concept art was created. As a result, far fewer drawings needed to be created, and the quality of the levels improved.

This is an example of reducing the waste of handoffs. By applying this practice to other handoffs, the team was able to create similar improvements across the entire production stream.

This is one example of dozens of changes the team made. By the end of production, it had improved takt time by 56 percent while significantly improving quality.

Knowledge, Skill, and Empowerment

Like Scrum practices, Lean practices create transparency in the production pipeline, which makes quality, velocity, and waste visible. One of the first things revealed is the variation of quality and velocity between separate teams. This visibility enables leads to know where to focus mentoring. In many cases, all that is required is to help a struggling content creator use a tool correctly or understand how to reuse assets to improve his effectiveness.

Lean thinking focuses skills on the final product, rather than on individual steps. Part IV, “Agile Disciplines,” discusses this topic in greater detail.

Many of the Lean tools described require teams to be empowered to make decisions and take ownership of their practices. In the previous example, the team decided to move the concept artist with the rest of the team. In many studios, teams are not allowed to make such decisions. It comes back to the lack of transparency. With transparency, teams are entrusted to make greater and greater decisions because of the performance metrics that ensure their decisions demonstrate higher productivity and quality. The difference in the number, quality, and frequency of beneficial changes increases as ownership of them is spread. Leaders might be skilled and insightful, but they can’t be everywhere. They need to rely on empowered teams to recognize and solve issues daily.

Outsourcing

Outsourcing has established its benefits for asset production. However, many studios have found that outsourcing limits the amount of iteration that takes place in the creation of large assets such as key characters or levels. This limited iteration impacts quality or introduces expensive rework that limits outsourcing’s cost benefits.

Lean production outside the game industry evolved to work with external suppliers. Suppliers to Lean companies need to become Lean themselves. Lean suppliers deliver smaller batches of parts to the main production line. This is done to allow quality improvements to be introduced more frequently and at lower cost.

How does this translate to game asset production? With our example, we don’t want to outsource the entire level production stream. The key is to outsource parts of the production stream that don’t require larger iterative cycles that should remain in the studio. For level production, studio teams retain large layout tasks and outsource the component assets used in these layouts. An example of this is environment sets, or collections of assets, common throughout a level. If a project needs a large city level, the team outsources all the props, such as light posts, mailboxes, vehicles, building components, ambient sounds, and so on. These environmental sets are brought into the layout steps (high-resolution art and audio layout). This enables continued iteration of the layouts at the studio.

Figure 10.15 shows a production stream with the environmental art outsourced.

The outsourced assets are identified in level concept and design to give sufficient lead time. These assets are delivered as they are developed rather than in a single batch.

Note

Many layout tools support late introduction of outsourced components. An example is the Unreal Engine 3 editor. The packaging system allows for levels to be laid out with proxy assets that are automatically replaced as the high-quality outsourced assets are delivered. For example, the studio artists could use blue rectangles in place of doors, and when the outsourced doors are delivered, a single instance change replaces them all.

Working with Scrum

When the project enters production, asset production teams may not use practices such as Sprint planning and tracking. However, these practices are still used by other teams to continue innovating new features.

Sprints are still valuable in production. Asset production teams demonstrate the assets that have been completed since the last Sprint review. Production teams don’t plan Sprints. Instead, they periodically fill a Backlog buffer with a set of ordered assets to work on next. The team selects a Backlog buffer size limit and fills it to that limit every time it is emptied.

Teams might also mix production and feature development work. For example, consider a level production team that has a few programmers adding effects, enhancing AI, and improving tools. The programmers plan typical Sprints with user stories and a Sprint Backlog, while the level creators use a Kanban workflow. Figure 10.16 shows how the two task boards are combined using a Scrum “swim lane” added to a Kanban board.

Retrospectives and Daily Scrums are still essential practices for production teams to address impediments and to improve how the team performs together.

What Good Looks Like

There is no project management approach that can turn a studio into a factory that mass-produces hit games that ship on schedule and budget. Some goals are impossible, at least with the constraints we’re given. The best a project management approach can do under those circumstances is to discover that fact as early and cheaply as possible.

I learned the “find the fun” and “fail fast” lessons for pre-production from Nintendo in the ’90s, but found that managing risk and content production needed far more planning than an iterative approach. This is what sets game project management apart from software project management.

Good is managing risk, protecting developers and the quality of the game from the pressures of hitting very specific targets on a very uncertain and speculative plan.

Summary

The additional challenge of stages with game projects doesn’t diminish the value of Agile or require complex plans or project management structures. It requires Product Owners to be aware of impacts that features have on production costs and for teams to adapt their practices as they enter production.

Additional Reading

DeMarco T., and T. Lister. 2003. Waltzing with bears: managing risk on software projects. Dorset House Pub.

Ladas, C. 2009. Scrumban: Essays on Kanban Systems for Lean Software Development. Seattle: Modus Cooperandi Press.

Liker, J.K. 2004. The Toyota way: 14 management principles from the world’s greatest manufacturer. New York: McGraw-Hill,

PMI. 2013. A Guide to the Project Management Body of Knowledge. Pennsylvania: Project Management Institute.

Poppendieck, M., and T. Poppendieck. 2007. Implementing Lean Software Development: From Concept to Cash. Boston, MA: Addison-Wesley.