Chapter 7
The Product Backlog
When we start working on a game, it’s natural to want to know what we are going to make. That perhaps sounds pretty obvious, but it takes effort to achieve. Without a shared vision of the goal, everyone would start developing a game with separate points of view.
But we often go too far. We create design documents that have hundreds of pages that describe the mechanics of the game and the levels and characters in great detail. If you’ve ever worked on a game that had a large design document written up front, ask yourself:
How closely did the finished game resemble the original design?
If the document evolved with the emerging game, was it a good artifact to communicate those changes (that is, a “living document”)?
Was the design document ignored after the game was signed by the publisher or stakeholders?
The Solutions in This Chapter
What we need is a planning method that does the following:
Communicates the order and value of the features: The team needs to focus on delivering the highest-value features ahead of the lower-value ones.
Enables change and communication of change: Change is going to occur, and discussions about those changes need to happen regularly and frequently.
Is a placeholder for future communication: Documentation can’t replace conversation. There must be a balance between the two.
Enables details to emerge as more information is learned: It shouldn’t require all design details up front or impose a great deal of effort to spread knowledge or change.
Delivers potentially deployable features in value-first order: Features are built in the order of the value they add for the players who buy the game. Delivering the highest-value playable features ahead of lower-value ones drives the development of the game rather than a preset ordered list of work. This enables the true value of the game to emerge earlier, when we can make better decisions about its direction.
Plan by feature, which creates cross-discipline behavior: Delivering features every Sprint encourages separate disciplines to collaborate to produce results that matter to the stakeholders. Problems that impact one discipline impact the rest and are likely to be solved more quickly rather than ignored for the sake of a schedule.
Avoid debt to allow better measurement of progress: Sprints combine the full cycle of feature development. This reduces the debt of debugging, tuning, polishing, and optimization and makes the pace of development easier to forecast. Establishing a Definition of Done with the Product Owner enables the team to understand each Sprint Goal and the quality required.
A Product Backlog addresses all these needs. This chapter describes Product Backlogs, a different form of planning, that addresses the problems with traditional forms of planning, such as detailed design documents.
A Fateful Meeting
I’ll begin this chapter with an example from my own experience. I recall a certain meeting years ago on a game project called Smuggler’s Run. Although I’ve attended hundreds of meetings like it, none other would change the focus and course of my career as this one did.
The purpose of the meeting was to review the work remaining to achieve the alpha milestone, which was two weeks away. Alpha was the cut-off date for incomplete features; features planned for but not fully implemented were to be cut from the game. You could sense how close alpha was. Smuggler’s Run needed to be out for Christmas. It was also a launch title for the PlayStation 2. Everything rested on this game being on time. No one really wanted to be at this meeting; there was much to do, but the meeting was critical and, as the director of product development for the studio, I had to know whether we would achieve alpha.
All the leads attended. I had to keep an eye on the lead programmer and lead designer; they often clashed over the ideals of design and the feasibility of what our technology could accomplish. They approached their roles perfectly, and the natural tension between them benefitted the game. I just had to make sure it didn’t go too far.
That day tested that tension. The lead designer had brought a big concern to the meeting. “When are the animals going to be added to the game?” he asked. The lead programmer didn’t have a clue about this feature. So, the lead designer pulled out the game design document and pointed to a paragraph on page 97, which described the feature. The game was to have animals wandering about the levels in herds that avoided the player vehicles. The feature had been added to the game design document partway through the project, and the lead programmer had not read it.
The lead programmer didn’t know whether to laugh or explode in anger. “How am I expected to know when a new paragraph was added to the design document?” he demanded. “I barely read it the first time!”
We spent some time calming everyone down and were finally able to discuss options for the feature. In the end, a simplified version of the feature was added. Rather than herds of animals avoiding the vehicles, lone animals wandered about ignoring the vehicles and, when hit, turned into tumbling rag dolls. As it turned out, this was a popular feature in the game. Players spent hours hunting down the entire populations of animals. Had this feature been more exploited, the game would have met with even more commercial success.1
1. And the ire of animal rights groups everywhere.
Why Design Documents Fail
As the project director, that meeting was a wake-up call about the issues of documentation and communication. I saw the futility of trying to know everything about a game up front and using monolithic documents to capture change and knowledge. This started my path to Agile.
The Smuggler’s Run game design document was well maintained by the designers, but it still failed in a few basic ways:
It failed to communicate the value of features: Each team member evaluated the value of each feature on their own.
It failed to communicate changes to the rest of the team: Few team members reread the document on a regular basis to keep up with the frequent changes.
The conversation that occurred two weeks before alpha was very fortuitous, but which features make it into the game shouldn’t be left to chance. What we had was a failure to communicate. The information in the document was valuable, but the document itself was a poor medium for communicating change and order.
This underscores some problems with the traditional requirements of gathering and tracking. One way to avoid these communication pitfalls is by using a Product Backlog, which is a Scrum artifact designed to provide planning centered around change and communication.
The Product Backlog
The Product Backlog is an ordered list of everything believed to be needed for the game. A Product Backlog:
Is written in “business language.” It’s focused on what is important and valuable to the player of the game or the user of a tool or pipeline
Is ordered, so that the team is always working on what is most important
Supports continual planning as the game emerges, so the plan matches reality
Accommodates frequent change and emergence in a low-cost way to allow better transparency and communication between the team and stakeholders
Encourages team engagement and alignment with the vision of the game by having it participate in defining and discussing the emergent detail
Is organized to reflect team structures, which reduces dependencies and the debt of unfinished work
Product Backlog Items
The features and other work identified in the Product Backlog are called Product Backlog Items (PBIs). Unlike detailed sections of a design document, PBIs briefly express the desired behavior for something in the game, tool, or asset pipeline from the point of view of the user of that functionality (for example, a player or artist). PBIs can describe any type of work:
Features
Content
Tools/pipeline work
Technical/architectural work
Bug fixes
Prototypes and experiments
Chapter 8, “User Stories,” and Chapter 9, “Agile Release Planning,” explore how to write PBIs that express the reason for their implementation and the value the implementation provides.
Ordering the Product Backlog
The Product Owner manages the order of PBIs in the Product Backlog. It’s done with the input of stakeholders, team members, and domain experts. The following guides help determine the order of each PBI in the Product Backlog:
Value: The value that a PBI adds for the player who buys the game is the main criteria in determining the order of that PBI on the Product Backlog. By focusing the team’s effort on adding the highest value every Sprint, Product Owners generate the best possible ROI. They measure value using their own judgment, feedback from focus testing, and feedback from stakeholders. Value applies to the “nonfunctional requirements” as well. Tool and pipeline PBIs that improve productivity have a place in the Product Backlog because improving productivity also improves ROI.
Cost: Cost is a key factor in the Product Owner’s ROI calculation. Some highly desirable features might cost too much to implement. An example of this is implementing fully destructible geometry for a shooter. Although this feature may add a great deal of value to the game, the Product Owner must weigh its value against the cost of developing the technology and producing level assets that leverage it. Two equally valuable features are often ordered by cost; the lowest-cost feature is given a higher order.
Risk: Risk implies uncertainty about value and cost. As a team refines its knowledge in these areas, the Product Backlog becomes a better tool for the Product Owner. PBIs with higher risk are often ordered higher to refine a Product Owner’s understanding of value and cost. If they are not, then potentially valuable PBIs might be left at the bottom of the Product Backlog and potentially dropped because of schedule or budgetary limitations.
Knowledge: Sometimes Product Owners don’t know the value, risk, or cost of a feature. They simply need more information. Early prototypes, experiments, and investigations will allow you to learn more about the value, cost, and risk of features being considered.
Note
This book uses the term ordered rather than prioritized when talking about the Product Backlog to reflect the idea that sometimes the order of work doesn’t match the priority. For example, a roof might be more valuable than the foundation of a house in a rainy environment, but the foundation still has to come first.
Continual Planning
The work done by a Scrum team is determined by the order of PBIs on the Product Backlog and what the team is capable of accomplishing. Every Sprint, a team pulls PBIs off the top of the Product Backlog and considers the development tasks to implement them. For this to happen, each PBI the team considers must be small enough to be accomplished in a single Sprint. Therefore, ensuring the PBIs on top of the Product Backlog are broken down enough is where most of the continual planning (usually about an hour a week) takes place.
Note though that not every PBI on the Product Backlog should be small enough to fit into a Sprint. If they were, then the Product Backlog might contain thousands of PBIs, which is too cumbersome to maintain. Instead, the lower-order PBIs are not broken down until the higher-order PBIs are completed. This is an advantage because the lower-order PBIs, which are more distant from implementation, are expected to change as the team and stakeholders learn more about what is possible and what is fun.
A useful metaphor for the Product Backlog is an iceberg (Cohn, 2008), as shown in Figure 7.1. The highest-order PBIs are represented by snowballs at the top of the iceberg, which are small enough to be completed in a single Sprint. Below them are the lower-order PBIs, or larger chunks of ice, called epics.
Figure 7.1 The product-planning iceberg
Everything above the waterline represents the work we want to complete in the current release. Each Sprint sweeps off the snowballs (PBIs) from the top of the iceberg. The stakeholders and team break down the larger chunks of ice (epics) into snowballs for the next Sprint.
Allowing for Change and Emergence
As we’re breaking out the detail of a game as we’re developing it, the Product Backlog never grows too large. Even the largest games rarely have more than 200–300 PBIs on them. Because of this, revisiting the Backlog every Sprint to make any changes based on what the emerging game is telling us doesn’t take much time. This is something we can’t do with 300-page design documents.
Encouraging Team Engagement and Alignment
Every Sprint, as teams complete PBIs at the top of the Product Backlog, they participate in breaking down larger PBIs further down the list into smaller PBIs that could fit in coming Sprints. This activity engages the team in conversation with the Product Owner and helps them participate in the decision making and understand the vision of the game.
Creating the Product Backlog
Unlike a traditional game where a large design document has to be created up front, an initial Product Backlog only has to have enough detail to get teams started. Typical PBIs on an initial Backlog could be:
Some prototypes to experiment with
Initial architecture dependencies
Basic camera, control, and character features
A small set of PBIs can be created in less than an hour in a simple workshop or in a Release Planning meeting (see Chapter 9).
Managing several hundred PBIs isn’t a significant challenge for a tool. Excel has been used by teams to manage their Backlog without difficulty. Commercial tools, such as Jira, are common as well.
When searching for a tool, there are a few features you may want to look for:
The ability to manage a hierarchy of PBIs: A Product Backlog is a hierarchy or set of hierarchies of epics that are eventually broken down into Sprint-sized PBIs. Often, individual branches of these hierarchies are owned by individual Scrum teams.
Allows sizing information to be assigned to PBIs: As described in Chapter 9, tracking size estimates of PBIs can be useful to forecast how well we are progressing towards target dates.
Ease of use with low-cost licensing: Having a tool that only one person can use at a time or is so difficult or costly for others to use can limit team collaboration.
Has good visualization options: Teams will often examine the Product Backlog through different “lenses.” Sometimes, they’ll want to examine one branch, and other times individual PBIs, and often the entire tree.
Different Forms of Backlog
I’ve seen and experienced many different forms of Product Backlog. Some worked, some didn’t.
Initially, we tried using index cards to store our Backlog. A physical Product Backlog had its benefits, but after a while, we felt nervous about having the Backlog for a $40 million game on a deck of cards that someone could easily leave in their pocket and send through the wash.
Our favorite form of Backlog was in a mind map. It represented the hierarchy very well, and using a 42-inch plotter, we were able to create large posters of epic branches for individual teams to hang in their area. Chapter 21, “Scaling Agile Game Teams,” explores mind mapping a Product Backlog in more detail.
What if our Stakeholders Demand the Big Design Document?
Few stakeholders (publishers, managers) still believe that a big design document ensures a good game will be delivered on schedule. Often the reason they still ask for it is due to a lack of trust; fatter documents make some managers feel better.
The way to build trust is to create transparency (see Chapter 17, “Working with Stakeholders”). Invite them to help build the Product Backlog and maintain it. Invite them to Sprint Reviews.
Building trust takes time, and they might not drop the demand for the detailed design right away.
Managing the Product Backlog
Even for the largest games, to stay manageable, a Product Backlog should never grow beyond several hundred PBIs in size. Live game Backlogs should be even smaller (see Chapter 22, “Live Game Development”).
Backlog Refinement
After a Sprint, teams will refine the Product Backlog based on the progress of the game. This is done in a meeting called the Backlog Refinement. In this meeting a team (including the Product Owner) will
Add any new PBIs identified in the last Sprint Review
Break any large PBIs on the top of the Backlog into smaller PBIs that will fit in the coming Sprints
Estimate the size of any new or split PBIs and update the forecasted Sprint Goals (see Chapter 9)
Eliminate older PBIs that are no longer desired or valuable
Teams often include some of the work done in the first part of Sprint Planning in the refinement meeting, such as discussing the potential goal for the upcoming Sprint. This can reduce the amount of time needed in Sprint Planning.
Who Attends the Refinement and When?
The Scrum Team decides when and where to hold a Backlog Refinement. It can invite anyone else, but keep in mind that a limit exists as to how many people can participate before conversation starts to slow down.
Teams hold Refinement sessions just before or after the Sprint Review. Although refining the Backlog before the Review might lack full stakeholder feedback, teams are often aware of what they’ve accomplished and find refining the Backlog before the Review a time saver.
Having the Development Team participate is useful. The conversations that lead to how the Product Backlog is ordered and how the Backlog items are created helps share the vision of the game with the team and leads to its making better decisions about achieving Sprint Goals.
The Product Owner can add or remove PBIs from the Backlog at any time. The Scrum Master has to ensure that the Product Backlog is being maintained and shared and that the Product Owner is not ignoring the Product Backlog.
Techniques for Ordering the Product Backlog
The order of PBIs in the backlog determines the order in which they are developed. It is a balance of cost, risk, knowledge, and value.
For example, while we might think that value to the player may be the sole factor that determines a PBI’s order, cost is often the limiting factor. Just as I might want to drive a Ferrari, my bank account and spouse might have something to say to discourage that choice.
The following are techniques commonly used to evaluate these factors.
Evaluating Cost
Ordering by the cost of a feature is often the most attractive to stakeholders because they can attempt to use numbers (such as time, people-days, and so on) to do so.
However, evaluating cost isn’t as easy as it might sound. Cost depends on the risk associated with a feature and the experience and skill of who is implementing it.
Chapter 9 explores the challenges and techniques of evaluating cost in greater detail.
Evaluating Risk
Risk is harder to quantify. In this case, like other hard-to-quantify elements, a variation of relative ordering (often referred to as a rank ordering) is useful. A risk matrix is a valuable tool for this.
A risk matrix is a simple tool. Each area of risk that has been identified is placed on a 2 x 2 map sorted, relative to other risks, by the probability of the risk occurring as well as its impact (cost or schedule) on the game (see Figure 7.2). The risks are then ordered by which quadrant they occupy on the map.
Figure 7.2 The risk matrix
Looking at the matrix in Figure 7.2, the risks in the “A” square are the ones we would be most concerned about. These could be risks like “we can’t find enough good programmers to hire and train in time.” Risks in the “B” squares are next. Those risks remaining in square “C” can usually be ignored. They are unlikely to manifest, and if they do, their impact will be minimal.
With a set of risks ordered by impact and probability, you can establish a plan to manage them. Chapter 10, “Video Game Project Management,” explores an approach for doing that.
Try Running a Premortem
A premortem (Brown S., Macanufo J., 2010) is an imaginary postmortem held at the start of a game. Participants put on their “future hats” and imagine themselves gathering a month after the game has been shipped. They then describe all the problems with the game and with the effort to develop it. These scenarios are used to populate a risk matrix and identify solutions long before these potential problems occur.
The Practice
Small groups of people (five to nine in size) gather to create a poster for the game. On this poster, they write the game’s goals (genre, market positioning, major features, and so on) and create the plan (critical dates, cost, team structures, and so on) on one half. The other half of the poster is set aside to capture “what went wrong.” Teams fill the “what went wrong” half with sticky notes, each containing one area where things went wrong with the game in the market or in the effort to develop it.
After a team completes the aforementioned steps (30–40 minutes), one member of the team presents their poster to other teams or stakeholders.
Tips
Here are some tips that can help when running a premortem.
If there are a lot of “what went wrong” sticky notes on a poster, ask the team to pick the three items that are most impactful to present.
Identify areas where differences exist in the goals and plan section. This is a great opportunity for a Product Owner to create and discuss a shared vision of the goals and plan for the game.
Evaluating Knowledge
One of the best ways to gain knowledge about something unknown is to perform an experiment. A useful tool for prototypes is using a timeboxed PBI called a spike, which limits how much time the team spends working on it before it can be evaluated. An example of this is a two-week prototype to determine whether the physics engine supports destructible geometry. If this spike demonstrates that the value, cost, and risk of implementing the system and toolset is not too great, the Product Owner is more likely to raise the order of a PBI to develop the full feature.
Evaluating Value
Knowing what your players want can be a challenge, especially the further out in time you try to forecast. Although live games can use Key Performance Indicators (KPIs) and frequent releases to guide value over the short term, identifying feature value over the mid- and long term can be nebulous. The following tools can help categorize value.
Kano Analysis
The Kano analysis model prioritizes features into the following market categories:
Delighters: These are features that are rare in other games or have never been seen before. They would be heavily promoted in a marketing campaign and would delight players; for example, massively persistent/destructible MMO environments.
Satisfiers: These are features that players are not surprised by, but enjoy. Competitors may or may not have all of these; for example, collaborative online zombie maps in a first-person shooter.
Basic expectations: These features are not advertised but are expected by the player, and their absence will upset them; for example, save game checkpoints.
Batman Quality!
“Product Backlogs are great for prioritization, but that doesn’t necessarily correlate to quality. But we’ve found a way to measure quality through the Backlog by adding a few more dimensions: customer impact and quality goal. Customer impact would be used to specify how much impact the implementation of the user story would have to the end customer, and quality goal would be used to set several expectations and alignment per quality level.
“You can measure customer impact at three levels:
Minimum is the minimum passable version of the story.
Awesome is making the story good or better than competition or user expectation.
Batman is something noteworthy and amazing, because, well, it’s Batman.
“Each has their Definition of Done, which establishes the quality goal. These metrics allow us to look at completed stories and measure the number of stories at certain quality levels, completed by customer impact. If everything is minimal, that gives us a good view of how it may be perceived.
“Having user stories with additional clarity of quality goals allows the ability to do a competitive marketing analysis and make more informed scoping decisions. It is extremely unlikely that any project can implement every user story at the highest level. But it’s also not necessary. By having the customer impact measure for items that are low impact, it is most likely safe to implement the minimum version.
“Before adding these measures, all user stories can be seen to be equal in impact and quality. Throughout the project, you can run a Backlog report; using the measures of customer impact and quality on completed features will give you the ability to view the current quality state. We can then put a higher weight on the high customer impact items and calculate a quality score for the product. Using this helps make better scoping decisions, to at least give us better odds at delivering the right quality, in the right places.”
—Brian Graham, Production, Playful Corp.
MuSCoW Analysis
MuSCoW Analysis prioritizes features into the following categories:
Must have: Features we cannot ship without. If we cut them, we’ll fail; for example, online gameplay on a first-person console shooter.
Should have: Features that can be cut, but would impact us if we do; for example, particle effects for a mobile slots game.
Could have: Features we would like to keep, but can cut without much impact to the game; for example, posting game progress on Facebook.
Won’t have: Features we will not keep; for example, a small character that follows the player around in the game incessantly giving them advice.
MuSCoW is especially useful for identifying an initial feature set for a live game with a minimum set of features. Its benefit over Kano is to explicitly identify features that won’t be included with the stakeholders.
Value/Cost Ordering
The matrix in Figure 7.2 can also be used to order PBIs using cost on the horizontal axis and the value on the vertical axis, as shown in Figure 7.3. PBIs are then ordered through the four quadrants:
High Value, Low Cost
High Value, High Cost
Low Value, Low Cost
Low Value, High Cost (don’t bother with these!)
Figure 7.3 The value-cost matrix
Defining “Done”
Each Sprint, teams commit to completing a number of PBIs from the Product Backlog and demonstrating that these are done in the Sprint Review. However, defining what done means can be challenging. There are many bad examples of what “done” means in the game industry, such as
“It runs on my PC.”
“It looks good in Maya.”
“It compiles.”
“I checked in the asset.”
These loose definitions result in debt piling up in the game. Let’s look at the types of debt that exist in video games, how debt is managed, and how Definitions of Done can help.
Types of Debt
Ward Cunningham, one of the authors of the Agile Manifesto, once likened the problems with technology, such as bugs and unmaintainable code, with financial debt—it grows the cost of payback over time.2
2. http://wiki.c2.com/?WardExplainsDebtMetaphor
Debt results in an ever-expanding amount of work that needs to be addressed before the game is deployed. Numerous forms of debt exist for game development, such as
Technical debt: Bugs; slow and unmaintainable code
Art debt: Assets such as models, textures, and audio that need to be reworked or improved
Design debt: Unproven mechanics waiting to be fully integrated into the game to prove their value
Production debt: The amount of content that the game needs to be commercially viable (for example, 12 to 20 hours of gameplay)
Optimization debt: Work waiting to be done to prove the game can run at acceptable frame rates on the target platforms
If unmanaged, the amount of work to address debt is unpredictably impactful and results in crunch and a compromise in quality as key features or assets are dropped in favor of maintaining a schedule.
Game Development Debt Example
One typical example of game development debt is in the creation of characters. Characters are often designed, modeled, rigged, and animated wholesale before we really know what we want them to do or before we know their budget. This is done because of schedule pressure or a plan that optimizes discipline allocation over the delivery of working assets.
As a result, by the time we figure out what we want the characters to do, we realize that the requirements for rigging and animation have changed, and instead of having to re-rig and reanimate one character, we have to do it for 20.
Demonstrating one character behaving the way we want in the game before mass-producing the remaining characters would have avoided this debt. If the character production schedule is a critical path for the project, then prioritize it as a risk and order the work that removes the uncertainty of character budgets and requirements early.
Managing Debt
The benefit of managing debt is to reduce the cost of development, avoid crunch, and avoid compromising quality. This is easier said than done due to the pressure of delivering quantity over quality. When we estimate the time to deliver scope, we don’t estimate fixing bugs, refactoring code, or even the overhead of iteration. That work has to be factored in to create the space to manage debt.
Note
I’ve never seen anyone write a task that said, “The bug we’ll find next Wednesday will take 5 hours to fix.”
That work is emergent and results from the quality bar that is established and refined between the Development Team and the Product Owner. That bar is called a Definition of Done (DoD).
After every Sprint, the Product Owner and the Development Team can refine the DoD. This refinement often takes place in the Retrospective.
When first establishing a DoD, start with the basics. An example is
“When added, this feature will not crash the game.”
Tip
Don’t change the DoDs too much every Sprint. Each change will impact development practices and take a while for the team to adjust to, and some DoDs may need to be reversed after exploring those impacts.
DoDs are added based on growing or newly identified areas of debt. For example, if the addition of new features drops the frame rate too low on a regular basis, the team might introduce a DoD that says:
“The game’s frame rate stays above 20 frames per second.”
Some areas of debt, such as slow frame rate, can be hard to track down, but it’s far easier to find a cause soon after the work that caused it is introduced.
Note
The goal isn’t to eliminate all debt. A certain amount of debt is reasonable to carry. For example, prototypes don’t have to have production code or deployable assets up front because we may be throwing most of them out. You just don’t want 50 percent of the code to be “prototype quality”!
Development DoDs and Stakeholder DoDs
DoDs result from a negotiation between the Product Owner and Development Team as they balance effort and quality. They define two categories of DoDs, which are described in the following sections.
Development DoDs
Development DoDs are white-box (internal) standards for the Development Team. They guide how code is written and assets are created; for example:
All functions have sufficient unit test coverage.
All assets conform to standard naming and budget conventions.
These are managed by the Development Team and guide work within the Sprint. For games with more than one team, make an effort to unify the development DoDs so there is a consistency of quality across the entire game.
Note
Development DoDs introduce additional work during a Sprint, which can measurably slow the pace of introducing features to the game, compared to the pace that was measured before they existed. This additional work saves time over the long term by reducing debt (see Chapter 12, “Agile Technology”).
Stakeholder DoDs
Stakeholder DoDs are external requirements, often called non-functional requirements, which are needed to satisfy market and publishing needs. Examples of these are
TRC/TCR and App Store requirements
Platform requirements
Performance requirements
QA and DoDs
The Development Team is responsible for ensuring that each PBI completed in a Sprint meets the DoD. Teams often recruit testers to help validate them as well by running the game on various platforms and verifying each DoD during the Sprint.
This was one approach we used to slowly transform members of a “QA pool” to members of Scrum teams, where they become more effective at helping teams improve quality (see Chapter 15, “Agile QA and Production”).
Sets of Done
Ideally, an Agile game should be “potentially deployable” every Sprint; live games can truly be deployable. For games that have not reached their first deployment, we should have a playable game that would demonstrate value to players every Sprint but would have some epic features that still require work.
As a result, teams establish sets of DoDs as shown in Figure 7.4. Each set encompasses a group of DoDs appropriate to the development maturity of an epic. For example, we might have a set of DoDs that a prototyped epic would need to satisfy, such as running only on a PC, not crashing, and exhibiting reasonable frame rates. Another set would incorporate the DoDs necessary to deploy an epic to players.
Figure 7.4 Sets of Done
Here is a simple example set of DoDs:
Prototype: Demonstrates potential value. Assets are for demonstration purposes, and the game only runs on development PCs.
Internal Demo: Demonstrates feature value and identifies areas that need to be polished before the feature can be shown internally. The game runs on target debug platforms with lower than shippable frame rates.
External Demo: Demonstrates demo quality (90 percent assets, not all TRC/TCR requirements). The target frame rate and hardware resource budgets are met.
Deployable (Ship!): Ready to be deployed. It passes all TRC/TCR tests and has no memory leaks. All assets are polished.
During Sprint planning, each PBI can have a different set of DoDs assigned to it. For example, they might be prototyping one feature while preparing another for deployment.
Note
Having the DoDs posted for the team to see is useful to remind the team during planning what the target is for each story.
Challenges
Here are some typical challenges that teams encounter with Product Backlogs:
They contain tasks: Sometimes Product Backlogs look like large Sprint Backlogs, which contain discipline tasks. The Product Backlog is meant to express business value, not development work. Otherwise, the Product Owner is unable to order it properly.
Too many PBIs: When a Backlog contains thousands of PBIs, it becomes impossible to refine it frequently enough to keep pace with the emerging game. This is often the result of making many design decisions too early.
PBIs are too detailed: PBIs are meant to be placeholders for conversation. When each PBI has a large document’s worth of detail attached to it, conversations don’t occur.
Dysfunctional Product Ownership
The “visionary” aspect is central to the role of the Product Owner. Regardless of the methodology used, if there is a bad vision or a vision that isn’t shared, the chances of making a great game are diminished. This is an area of vulnerability with any project and no less so with Agile projects.
The advantage that Scrum teams have is that the visionary role of the Product Owner is more transparent and defined. A dysfunctional Product Owner should be quickly identified. Teams demand communication and Scrum Masters hold visionaries accountable for serving that demand.
Where is Matt Damon?
One of our first Agile-run projects was Bourne Conspiracy, which was based on the Jason Bourne franchise, written by Robert Ludlum and the basis for the successful Bourne movies starring Matt Damon.
We were given the job to create a game that followed the first movie’s plot. This was a challenge for us, because the game needed to ship when the next movie was released, on the next generation of consoles using an engine that hadn’t ever been used on a console. Added to that, we had never made a third-person action-adventure game. There was a lot of risk!
Our stakeholder wasn’t familiar with Scrum and so to build trust with them, we asked one of their executive producers to be the Lead Product Owner for the game (see Chapter 21). This was a big mistake.
Our Lead Product Owner didn’t visit the teams very often, and when he paid attention to the Product Backlog, he described a game that felt more like Splinter Cell than what you would expect from a Bourne game; Jason Bourne should use objects around him, such as rolled-up magazines, to defeat enemies rather than the high-tech gadgets in Splinter Cell .
We just collectively shrugged and focused on the technical and schedule risks of the game.
Unfortunately, the Ludlum estate, which had final say over the game’s release, wasn’t going to ignore this. It also turned out that Matt Damon wasn’t interested in participating either.
Unfortunately, our Lead Product Owner wasn’t communicating with these important stakeholders. As a result, we had to rebuild the game when the Ludlum estate rejected it, and we had to scramble to find a new lead actor.
Missing the movie’s release, not having Matt Damon, and the impact on quality from last-minute changes killed any chance of success. We learned a number of lessons about dysfunctional Product Ownership (described later in this chapter).
I’ve seen many examples of dysfunctional Product Ownership over the past decade. This section lists the most common dysfunctions and ways to address them.
The Proxy Product Owner
A proxy Product Owner is an underpowered Product Owner who stands in for the actual Product Owner. This can be due to many reasons, including:
The actual Product Owner is too distant or too busy to interact with the teams on a daily basis.
There is no one with the authority or accountability of a Product Owner, so one is created in name only.
There is a group of people to instantiate the Product Owner role, rather than one, and they need a single voice to speak with development.
Anytime an additional layer of communication exists between developers and decision-makers, it impacts effectiveness. Product Owners must have the authority to make calls with the accountability that they are the right ones. A proxy Product Owner can lead to disengaged teams due to a lack of a shared vision or decisiveness.
On the business side, the project can suffer from a lack of transparency. The different areas of focus can lead to delayed integration and delayed knowledge of what makes a game good or not.
Proxy Product Ownership can be hard to address because the roots are cultural and the shift to a single voice that speaks to the vision and project management aspects of the game will
Threaten those who currently occupy those aspects of the role
Be hard to fill because very few people are good at being both a visionary and project manager
Product Owner Committees
Product Owner Committees form when there are many equal stakeholders who hope to prioritize the Backlog through collaboration or when there is fear of elevating one of their own into the role of the “single voice of the game.”
The main potential dysfunction seen with this arrangement is from the team’s point of view. The feature POs or teams often don’t know who to speak with, and the committee members usually have a different perspective. Often, a release plan from a committee is the combined set of all desired features from the committee, which leads to long release cycles.
Example: A Product Owner Committee prioritized hand-to-hand combat as the highest priority epic for the next release of a game. The team invited one of the committee POs to its first Backlog Refinement meeting in which it discussed the design options in breaking the epic down. The only member of the committee that showed up was Amy. In the discussion, Amy emphasized that the hand-to-hand combat had to “look cinematic” at all times. This led to the team focusing on ways to avoid animation popping and adding more transitions. At the first Sprint Review, one of the committee POs, Dan, complained that the fighting was responsive enough and that the team should focus on the movement mechanics, such as jumping and running. A third committee PO, Steve, thought that multiplayer online needed more work and that the team should work on that.
The Dirty Dozen
I once worked with an MMO that had a committee of 12 Product Owners. Because they couldn’t narrow down the goals for the next expansion pack, it had to have all of their required features. As a result, it took well over a year to release each expansion pack, which frustrated their players.
The solution was to run an exercise called “Buy a Feature” (Keith and Shonkwiler, 2018), which helped reduce the set of desired expansion features to approximately six months of effort.
The main issue with transitioning from a Product Owner committee to a single-voice lead Product Owner is to address the concern that prevents it. The other committee members might fear a loss of influence or prestige if it’s seen that one of them is being elevated over the others. To overcome these fears, the lead Product Owner’s responsibilities must be communicated and demonstrated (see above) to the others in the committee. Coaching is often necessary to reinforce that the lead Product Owner acts in an inclusive way, and the other members of the former committee do not lose their voice.
Silo Product Owners
Silo POs are similar to a committee POs in that there is no single voice, but silo POs are a bit worse: These POs don’t even talk the same language. They focus on the underlying means to build features and not the outcomes. Silo POs usually result from adopting the Product Owner role in a siloed or matrix-driven organization where design, art, and technology department leads run the studio. As with a committee Product Owner, it was easier to assign the Product Owner title to the existing leadership and ignore the need for a single voice.
Silo Product Ownership can create confusion among teams and leave a lot of “loose ends” with the game. This can demoralize a team that doesn’t understand the vision because, regardless of whether a vision has been shared, if the prioritization doesn’t match the vision, the work will not align to it.
As with the Product Owner committee, one of the silo POs has to step up to be the single voice for the team. This can also raise concerns about roles.
Example: Project Zeppelin, a mobile game, has two Product Owners who are the founders of the small studio creating it. One is a designer, and the other is a programmer. Several teams are developing a World War I flight simulator game, which is the sole game being worked on at the small studio. When planning and executing sprints, teams will get a different set of priorities from their Product Owners. The design Product Owner will want to try out different mechanics and prioritize prototypes to the top of the Backlog during Sprint Planning. The technical Product Owner will prioritize technical risk reduction work to the top of the Backlog for teams that invite him to their planning sessions.
In the preceding example, with two founders providing guidance to the one project the entire studio is focused on, the founder who doesn’t take on the Product Owner role might be feeling diminished or threatened. As a coach, I would remind him that a Product Owner must rely on close collaboration with his peers to make the best decision about the prioritization of the backlog. Let’s say the designer became the Product Owner and began ignoring all the technical input for reducing risk. This points to a dysfunctional relationship between the two founders. This was the real dysfunction that instantiating the Product Owner role has exposed. It’s probably always been there and has been a threat to the studio from the start. A coach can work with these two to establish better trust and communication and help them fix the true root cause of the problem.
Attention Deficit Product Owner
I have heard some POs claim, “We need to focus on everything!” It’s great when a Product Owner has a strong vision. It’s not so great when that vision changes daily. Some famous game designers have been known for this. They have a dream one night, or their cat does something interesting that sparks an insight, and so on. Next thing you know, they are running through the studio excited about their new insight and impatient to see it being implemented.
Enthusiasm is great, especially when it can be shared, but it needs to be combined with a focus on demonstrating fun in the game. Focusing on too many things that will somehow come together months later is a debt we don’t want to owe.
Example: Nearly everyone has heard of the Duke Nukem Forever development story.3 The sequel to the hit game Duke Nukem 3D took 15 years to develop due to a continually changing series of goals, in addition to changing publishers and so on. Although it wasn’t a Scrum-based project and probably didn’t have a Product Owner, per se, it’s an extreme example of how a lack of focus can impact a game’s development. Teams with a wandering focus waste a lot of money and cause stakeholders to cancel projects.
3. http://en.wikipedia.org/wiki/Development_of_Duke_Nukem_Forever
A continually changing focus can have a bad effect on the team. It can lead to disengagement among the team members: No matter how engaged they get in a vision, it’s going to change, so why bother? Also, the sense of urgency that comes with these changes is not sustainable, which leads to poor quality decisions (reduced testing, refactoring, tuning, and polishing).
Additionally, this constant change of focus will have an impact on the velocity of the team over the course of a release and beyond.
The Scrum Master needs to intervene with Product Owners to help them understand the impact the constantly changing focus is having on the team. The main barrier is that this is the Product Owners need to “be heard by the team.” They can see the Scrum Master’s intervention as a threat to their authority. Therefore, the Scrum Master must take care to ensure the ideas of the Product Owner are being collected, ideally as user stories, to be addressed. The best place to do this is in a Backlog Refinement meeting. This is where the Product Owner can discuss insights along with development impacts and costs with a smaller group of experienced developers. It will also create a buffer between the developers who are focused on the current Sprint’s goal, because any decisions made during the Refinement meeting will only impact future Sprints.
If these changing goals impact release targets, the stakeholders must be informed. The Scrum Master has to ensure that they are aware of this and have discussed it with the Product Owner. Stakeholders who are engaged with the decision-making process are much less likely to make drastic decisions about the project.4
4. Ask any Duke Nukem Forever developer.
Tunnel Vision Product Owner
Bill Gates supposedly claimed in 1981 that 640K ought to be enough for anybody. No vision is perfect, but sometimes the hardest person to convince of that is the visionary. Given that a shipped game rarely resembles the original detailed design document, an equally detailed and prophetic vision buried inside someone’s head can be just as capricious. A common result is that the emergent gameplay quality is not taken into account and teams become disengaged.
I call this dysfunction “tunnel vision” Product Ownership. It’s common in studios transitioning from a design-document approach to an Agile one. Although the visionaries admit that detailed design documents fail to create a shared vision, they haven’t yet adopted an Agile mindset, which requires
Frequent face-to-face engagement with developers
The ability of teams to produce iterations of an emergent game that validates the vision (or not)
The ability of Product Owners to evaluate their vision against the emergent game
This leads to many patterns of dysfunction, as the example illustrates.
Example: The Product Backlog becomes a component list of features and functions necessary to fulfill the vision…someday. This reminds me of someone building all the parts for a prototype car. They build a carburetor, then the tires, and so on. Eventually, when all the parts are built, an assembly phase is entered. Even if all the parts fit together, it rarely results in a very functional car.
The same dysfunction happens in games. The following dysfunctional practices are seen with a tunnel vision Product Owner:
There are too many component (or functional) teams; for example:
AI team
Animation team
Design team
Graphics team
This is not to say that component teams are all bad. They are often necessary, but having the majority of a project composed of component teams usually results in delayed integration and knowledge (that is, debt).
Crunch Sprints (usually called something less obvious) occur with lots of overtime, little velocity, or just to tackle user stories created to address debt (refactors, bug fixes, polishing, and so on). If most user stories don’t read like features users would pay for, it’s generally a strong indication that a good Definition of Done is not in place, teams are doing a lot of component work, or debt is piling up.
Teams that don’t have a tangible vision or one that is months away from being realized have a hard time engaging in their work. This is especially true on larger projects. Nothing will drag velocity down more.
As with any cultural issue, tunnel vision is hard to fix. Transforming a detailed design document–focused culture to one of frequent communication and inspection doesn’t happen overnight. There needs to be some bridgework between the two. Before the documents are shrunk, a Product Owner needs to gain skill in communicating vision and in learning to respond to the emergent game. To start with, find ways to improve communication in Sprint Planning, Reviews, and Retrospectives. Establishing a better Definition of Done, and phrasing the Sprint Goal in terms of true value (for example, “What are we going to be able to play at the end of the Sprint?”) sets the context for what is expected (in the playable game). Focusing the Sprint Review on the state of the gameplay sends that message as well. For the team, it’s easier: Having a Product Owner responding to gameplay communicates priorities.
Having a vision is better than having none at all, but a vision must be tempered against reality.
Distant Product Owner
Distant Product Owners are physically distant from the people making the game. They could have the best vision for the game and be the creative soul, but because they are not near the team, they don’t get the same traction with the game. This is a very common dysfunction with large, distributed teams. The Product Owner can only be at one place at one time, and so all but one of the development locations will suffer from a distant Product Owner dysfunction.
Example: This was one of the problems with the Bourne Product Owner. When a Product Owner is distant, teams drift and stakeholders lose their voice. Each fall back on their assumptions about the game or lose sight of any vision.
If development is occurring in one studio, the solution to this dysfunction is to get another Product Owner. This can be a challenge due to a perceived loss of influence from the existing distant Product Owner (especially with a licensed game), but it has to be made clear to the distant Product Owner that his influence is already diminished from his distance. One solution is to create a developer-side Product Owner who has a good working relationship with the distant Product Owner, whom I’ll call the business-side Product Owner. This relationship works with a publisher/developer relationship where many of the marketing and funding decisions are made at the distant publisher site.
The developer-side Product Owner will own the Product Backlog and be able to make decisions about it but will also be able to develop a shared vision with the business-side Product Owner. The business-side Product Owner will have the responsibility of maintaining a shared vision with the marketing, franchise, sales, accounting, and executive stakeholders local to him.
Care must be taken that the developer-side Product Owner does not become a proxy Product Owner.
For large, distributed teams, please refer to the distributed development section (Chapter 21) for advice on avoiding the distant Product Owner dysfunction.
Ditching Product Owners
CCP Games, the developer and publisher of EVE Online, was founded in 1997. After the transition to Agile and Scrum in 2009, the EVE Online team at CCP had a fairly mature organization. One of the significant adaptations it made was the elimination of the Product Owner role and the shifting of that responsibility to the teams themselves. Andie Nordgren, former Executive Producer of EVE Online, describes the transition.
“Agile practices were seen as a given, and work on process was mostly about optimizations rather than any challenges to the form itself—a “mostly Scrum” setup with cross-functional teams with designers, developers, and quality analysts collaborating and taking responsibility for their work all the way into production. There were monthly release trains and smaller patches in between.
“The investment in cross-functional teams was a clear direction we wanted to stay with, and the problems at the time that disrupted these teams were all about how teams were directed and managed. At the time, teams had been cross-functional for a long time, but people management was still done matrix-style by work role: Managers for Engineers, Producers, Designers, Artists, and QA, respectively. For each discipline, there was also a director, and each team had a Product Owner, normally a Producer filling that role for one to three teams. It was a mix of team-based, discipline-based and individual-based practices that created a number of disruptions that hindered teams from performing well—mostly because individuals on these teams just had too many people telling them what to do. Also, Product Owners were hard to find. They ended up being Producers who could easily handle the production-style tasks of Backlog and stakeholder management, but lacked the knowledge of the users and the game to really make product decisions themselves, or they would be designers who could make good calls on game features, but didn’t have time to do so, because they were (often badly) managing the Backlog and the stakeholders. Either way, teams often felt victimized by both managers, directors, and their Product Owner, who—by the rules of Scrum—had the power to prioritize and make decisions about what the team should work on, and in practice represented the team in many key product discussions, when the actual product knowledge sat in the team. Frustrations were everywhere.
“With the compass pointed towards a structure that would support whole teams rather than individuals, we made two changes: First, we created a Development Manager role that was responsible for staffing a team and for its processes and work practices and the team’s deliverables once they were committed. From a user value perspective, these managers led from below—supporting the team that knew what user value to build, with the right resources and process to do so.
“For Product Ownership, we decided to move the role to the team as a whole, and ask every member to step up to take responsibility for what they were building and why: engineers, QA, designers, artists collaborating to define and build features with process support from their development manager. The responsibility would no longer sit with an individual outside the team, but with the team as a whole, driving home the point that everyone on the team was needed and a crucial part of how to deliver value to the players of the game. In practice on gameplay feature teams, designers often led the work on what to build but they now didn’t need to drown in Backlog management to do so, as they had that support from their Development Manager.
“This created a new type of commitment from teams and also a more empowered autonomy. New types of team collaborations emerged as teams realized they truly owned their own time and could make commitments to other teams. But it also became harder to coordinate and collaborate between teams as teams became reluctant to ask other teams for work or sign themselves up for work with a lot of dependencies on other teams. Eventually, some more explicit process for cross-team planning and commitments was brought in to counter these tendencies. Another gap was that a number of tasks and skills normally performed by Product Owners didn’t get the appropriate amount of staffing on teams. To ditch the Product Owner role, teams should have had embedded people or access to better data analysis resources and user research resources.
“Overall the change recognized that with entertainment products the uncertainty of what will work for users, and the fact that you can often not just ask them what they want, creates a product development environment less suitable for having a single person represent the customer needs. And in that case, having product ownership move from an individual to being a responsibility for the whole team working to deliver value to users might be a better fit.”
What Good Looks Like
A good Product Backlog:
Is easily accessible by any member of the team
Contains PBIs that express the reason and value of the work
Contains no more than 200–300 PBIs
Is refined regularly
Is ordered by a single voice: the Product Owner
A good Product Backlog reflects an alignment with the vision of the game between stakeholders and developers alike. It’s one single list of ordered work rather than separate lists of discipline work that is derived from a design document written at the start of the project.
Summary
The biggest cultural challenge in adopting a Product Backlog is the shift from writing detailed specifications up front to leaving questions unanswered until more is known. While there is great comfort for stakeholders in reading exacting specifications about all proposed features in a game and the details and cost of how they will be implemented, those are usually guesses, and first guesses on uncertain features are usually wrong.
It’s fine to document things we are certain about, but we can’t document away uncertainty. Uncertainty has to be executed away. We have to focus the team on the goal and work with it closely to find the best path to the best outcome.
Additional Reading
Cohn, Mike. 2006. Agile Estimating and Planning. Upper Saddle River, NJ: Prentice Hall.
Gray, D., S. Brown, and J. Mancuso. 2010. Gamestorming: a playbook for innovators, rulebreakers, and changemakers. Beijing: OReilly.
Keith, C., and G. Shonkwiler. 2018. Gear Up!: 100+ ways to grow your studio culture, Second edition.