Chapter 15
Agile QA and Production
Part of the role definition for quality assurance and production is the ability to communicate with all disciplines. Because fostering a common language across all disciplines is a principle for Agile game teams, quality assurance (QA) and production have an advantage when Agile is adopted.
At the same time, Agile practices change the role of QA and production the most. Fortunately, the change is for the better. This chapter addresses those changes and how the QA and production roles fit into Agile teams and organizations.
Agile QA
When my sons were in grammar school, I was invited to parent career day. It was always fun. To the class, “video game developer” was the coolest career imaginable. I could be standing next to the “astronaut/secret agent” dad wearing his space suit, and the kids would climb over him to ask me questions.
An inevitable question was, “How can I be a game tester?” They dreamt that the role of a tester is eight hours a day of the same fun they experience playing games. What could be better than that? Unfortunately, I was forced to destroy this notion. I describe the long hours with games that crash. I described how testers are often very frustrated because they understand what the game is and have very little influence over how it is developed. I don’t want to discourage future generations of testers; I just don’t want to give the impression that there is this wonderful job waiting for them simply because they are “good at playing games.”
As you may guess, the QA role attracts people who are passionate about games. The sheer number of people applying for this role gives the industry the ability to choose from among the best. We need to leverage their passion and experience far more.
The Solutions in This Chapter
In this chapter, we’ll learn how to improve and expand the role of QA on an Agile game team. We’ll also explore the role of producers and how their job potentially maps to either a Scrum Master or Product Owner role.
The Problem with QA
Traditionally, quality assurance is largely performed at the end of the project by a dedicated QA team. Figure 15.1 shows the pattern where most bugs are discovered on such a project after alpha, when all features are considered complete but not debugged.
Figure 15.1 Bug discovery rate on a traditional project
The reason for this is simple—testing is performed on a potentially shippable game. Because the typical game project does not achieve this state until post-production, most testing has to be compressed between the alpha and ship dates. As a result, armies of testers are hired in hopes of achieving a ship date with a working game. Hiring and properly training a small army quickly is impossible, so testers are minimally trained for each game.
This type of quality assurance doesn’t always assure the highest quality. By alpha, the important decisions about the game have been made, and the job of QA is to find and report minor defects. Most major defects are rooted too deep in the design or architecture of the game to be properly addressed. For example, if level pacing isn’t fun, it may be too late to correct it. This compromises the quality of the final game.
The other problem with traditional testing practices is that quality becomes the responsibility of remote1 QA and not the developers. Studio cultures encourage this by basing employee performance evaluations on the pace of feature implementation or asset creation. Bugs that do not stop progress are considered part of making the game and are tolerated before alpha. Adding QA practices during this time slows feature and asset creation in the short term, so it is deferred.
1. Such as a publisher’s QA department in another location or a QA service
Black Box and White Box Testing
Testing can be divided into two types: black-box and white-box.
Black-box testing uses a player’s perspective to test a game at the highest level. A tester is given a list of expected behaviors and the conditions or input for causing them. The tester then ensures that they occur. It’s also meant to ensure the pacing and mechanics are fun and engaging (see the “Play-Testing” section later in this chapter).
White-box testing uses an internal perspective to test a game. The components of the game, from code to assets, are examined and tested for standards and functional compliance. This requires skill and experience in the discipline that created the code or asset to be tested.
Most QA Is Just QC
Over the past decade I’ve seen a few examples of proper QA being done at studios. Proper quality assurance is a role that focuses on finding ways to improve practices that reduce quality issues. This is opposed to quality control (QC), which is finding quality issues after they’ve been introduced.
Typical QA practices I’ve seen are:
White box testing to ensure unit test coverage
Writing and managing test scripts to automate gameplay
Finding ways to automate manual testing tasks
Sometimes, people in these roles are called “product assurance engineers.” This is obviously a more skilled and expensive role than traditional QA role, but the advantages are
You need far fewer of these people than you would with an army of QC testers.
Fixing bugs is far more expensive than finding ways not to create a bug in the first place!
Agile Testing Is Not a Phase
The approach to QA on an Agile project is different from that of a traditional project in a number of ways:
Testing takes place on an Agile project throughout its life cycle: This is a major benefit of creating a potentially deployable version of the game every Sprint. Defects are not deferred or tolerated.
Testing is white-box testing: Testing focuses on the internal components of the game rather than the game as a whole. This requires more skilled product assurance engineers who can specialize in areas such as code reviews and test automation.
QA is part of every role: Each developer performs QA in his work. QA doesn’t eliminate this need but catches problems that slip through and helps improve team practices so that similar problems are more likely to be caught by the developer in the future.
Employing this approach evens out bug discovery rate to a consistent level throughout development without the necessary staffing up and down of QA.
Definition
The game industry traditionally refers to members of a QA group as testers. This title becomes obsolete as a studio adopts Agile. Testing becomes part of everyone’s job, and the QA function specializes and overlaps those of other disciplines. However, because everyone uses the term testers, I use it here along with QA, which defines the role more than the individual.
The Role of QA on an Agile Game Team
Compared to traditional approaches, Agile projects extract more value from testing. Much of this value comes from each team’s goal of delivering a playable and potentially shippable game every Sprint. The QA role shifts to take advantage of this:
Testers are members of Scrum Teams, rather than gathered in separate QA pools. This increases the speed of feedback on defects to the developers.
Product Backlog Items (PBIs) require QA action for approval before they are considered done by the development team.2 This creates a true pace of development, which should include fixing defects and reduces the cost and added risk of fixing them at a later date.
2. The Product Owner has the final say, however.
QA’s voice is heard throughout development, not just at the end. This allows its valuable feedback to influence the game.
QA’s role on an Agile team grows beyond its traditional role. It starts with QA’s participation in Sprint Planning. As each story is discussed, QA needs to understand how that story will be approved, before tasks are estimated. This usually takes the form of how the story and the Acceptance Criteria (AC) are written.
Consider, for example, the following story:
As a player, when I hit the jump button, my character will jump.
The team discusses this story and may ask the following questions:
Will the character have jump animation? Will it need smooth transitions?
Can the character jump to a higher level?
Can the character jump from a moving platform?
Can the character jump while walking or running?
These questions help the team understand the tasks necessary to implement and test the story to ensure it is done. For example, the answers to the previous questions may define the following AC:
The character will have a simple stand-in animation that will not be blended.
The character will jump up to a ledge or down to a lower level.
The character is not expected to jump correctly from a moving platform.
The character can jump from any starting motion, but its momentum will be ignored. It can jump only a fixed distance and height.
Using these AC, both QA and the developers verify that the story is complete before the end of the Sprint.
QA should also locate problems that are not part of specific AC but are part of development. For example, if the test level is missing some physical geometry that enables the player to “fall through the world,” QA should identify it as an impediment.
A tester should help to ensure that the team is not impacting other teams. This includes the following:
Performing regression tests on areas the team is working on that could break other areas of the game: For example, if the team is working on character control for jumping, keep an eye on other character motions to ensure that some side effect of the team’s changes is not breaking them. Another example would be if every time a player dies, his character assumes the jump pose, then the team working on jumping has probably broken something!
Testing tools and pipeline changes that a support team wants to release to the content creators: This includes testing beta versions of a tool with the content creators to verify changes have not broken a workflow.
Finding ways to improve testing: For example, if bad textures are frequently crashing the game, QA should raise a request to have the texture exporters improved to catch the problem.
Some of QA’s time is spent verifying gameplay and offering advice on areas of improvement. This requires an ongoing dialogue with members of the team. Because QA is a voice for the consumer, it needs to keep a “consumer eye” on the game. It should point out any issue that the consumer will notice (such as progression stoppers).
QA will be challenged in this new role. People in this role are required to communicate with every discipline, understand the tools, and stand up and be heard. QA has traditionally been a “gateway role” for other roles in game development, so developing these skills is appropriate to grow their career in game development.
QA, Embedded or in Pools?
To what degree should testers be spread across teams or located in a QA pool that tests independently? There is no simple answer beyond “It depends.” It depends on what the team and project need in order to ensure that quality is built into everything being worked on. It depends on the skill level of the testers and how much white-box testing they can accomplish. It depends on the testing tools, harnesses, and telemetry that are available.
When a game is in early development, the depth of gameplay and limited assets don’t need as much separate testing; teams might share testers. As the project progresses, there is more to test. The game may need to run on several platforms. The growth of complexity requires not only more testing of the added features but also more regression testing to ensure that previously added features have not been inadvertently broken. QA pools can even out this workflow, helping team testers in times of need. As a game approaches its ship date, QA may be concentrated in a single pool or in separate pools dedicated to each platform.
A tester embedded within a single team takes on a more expanded role. He can assist the team in addressing impediments and completing tasks. For example, if an artist needs to test an asset on the iPhone, an embedded tester can help by creating an iPhone build, launching it, and calling the artist over when the asset is visible. Embedded testing encourages a tester to build his white-box testing skills and knowledge to overlap his work with the other developers on the team.
How Many Testers per Team?
A frequent question raised is, “How many testers does a Scrum team need, and when should they be added?” It’s the same as asking how many animators or audio designers a team needs; the answer is “as many as you need!”
The number of testers depends on the testing needs of the team. This hinges on the following:
The Definition of Done: As the Definition of Done for Sprints approaches the final shippable state, more testing is required. For example, if a Definition of Done requires the game to be completely playable from start to finish without crashes on any platform, more testers are required than in early development when the game has to run on a development PC for a few minutes at a time.
Team testing practices: If the team uses practices that support higher quality, such as Test-Driven Development (TDD) or thorough export testing, there will be fewer problems that QA is needed to find.
Test automation: It depends on the coverage level of automated test utilities, ranging from simple smoke tests to complex scripting, that allow the game to “play itself” and offload manual testing.
Playability testing needs: The game needs hands-on, in-depth testing from a player’s perspective. QA needs to evaluate usability, pacing, and difficulty levels and provide feedback to the team.
Typically, one tester per team is enough for most of the game, but ultimately the team’s needs are driven by Sprint results. As the stakeholders raise the Definition of Done, it may put pressure on the team to recruit more testers to help it meet its goals.
Using a Bug Database
As discussed in Chapter 10, “Video Game Project Management,” a debugging stage after the alpha date is unavoidable for many games. For these projects, the QA staff is expanded, and the entire game is tested from beginning to end.
It’s dangerous for a project to rely too much on post-production testing and allow significant defects to remain undetected throughout production. When such defects are uncovered after alpha, they can invalidate a great deal of work. For example, discovering that the disc budget has been exceeded by 50 percent in post-production is very bad.
Ideally, post-production efforts should focus on tuning and polishing tasks that are identified and prioritized daily. Even without major flaws, tracking polishing, tuning, and minor bug-fix tasks can overwhelm a Sprint task-board approach.
It’s important that there is only one Product Backlog for the game and that the Product Owner prioritizes a single list of work for the team to draw from every Sprint. Before alpha, any bugs that are addressed must either come from the team during the Sprint or come from the Product Backlog. If a bug database is maintained prior to alpha, then any bugs that are to be worked on are moved from there and placed on the Product Backlog to be prioritized by the Product Owner.
At alpha, all unimplemented stories on the Product Backlog are cleared off (they may become part of the Product Backlog for the next version of the game). From here out, only bug fixing and tuning work can be placed on the Product Backlog. The Product Backlog might even be replaced by the bug database.
The Product Owner manages the priority of bugs with QA through frequent triage sessions. As with hardening Sprints, described in Chapter 9, “Agile Release Planning,” the team draws a set of tasks to work on daily, without a specific Sprint Goal. The new goal is to burn down all the high-priority bug fixes and tuning work.
Advanced Practice: The “Done Done” Column
Although quality assurance is every developer’s responsibility, the depth, complexity, and interconnectedness of games means that they benefit from someone playing the game during each Sprint to catch unanticipated interactions and behaviors.
The “done done” column is added to the right side of a Sprint task board, after the traditional “done” column. When a PBI in the Sprint Goal meets the Definition of Done, it is placed into that column. This creates a more concrete, visible declaration of which part of the Sprint Goal has been completed.
Play-Testing
A major benefit of producing a potentially shippable version of the game every Sprint or release is that a game is tested early by potential consumers. Play-testing is a practice where consumers are lured in to play a game in development—often with the offer of free pizza—and provide useful feedback about the gameplay experience.
Play-testing can be as informal as a conversation. It can also take a more scientific approach by recording progress metrics or the answers to carefully designed surveys.
Play-testing has substantial, but limited, benefits:
It reinforces a Definition of Done: Developers, who see a game every day, overlook flaws or shortcomings. Play-testing is often a shock when they observe how these problems impact real consumers. For example, that missing piece of collision geometry is always found by play-testers!
It won’t produce any breakthrough ideas: Don’t expect the play-testers to provide original ideas. That’s the developer’s job. At best, they provide feedback that can improve Backlog prioritization.
It teaches about usability and challenge: Ever see a player simply run past a painstakingly scripted boss? It’s a sobering thing to see. It’s also very frustrating to see a player walk against a wall for two minutes until you realize that she is trying to make sense of a HUD map that is too vague!
Experience
Play-testing is a tool for focusing a team on done. We often scheduled play-testing sessions for the end of a Sprint. This made the team look at the game with a different set of eyes and uncovered many polishing and tuning tasks that had previously been invisible to us.
Play-testing is often organized and run by QA or usability specialists. These duties include the following:
Recruiting play-testers: Local universities and game retail shops are teeming with good play-test candidates. Be sure to enlist people who represent the full scale of demographics and skill levels. For example, a hard-core gamer won’t provide the same feedback about the tutorial level as a casual gamer. Maintain a database of people, and invite the more valuable play-testers back.
Organizing and running the session: A poorly organized and run session elicits inferior feedback from play-testers. Don’t waste their time.
Including the developers: Having the developers see players interacting with their game in ways they did not anticipate is very beneficial. It can lead to improved interfaces and usability (see the sidebar “Damage Meter”).
Letting play-testers meet the other developers: Many testers are curious about game development and relish the opportunity to speak with the rest of the team at the end of the session. Encourage this. The conversations can be very fruitful!
Publishing the results: QA’s observations of how well the game was received provide many insights. However, care must be taken not to “overinterpret” the results to include subjective bias from QA. A simple compilation of the answers helps the team understand the results. Leave the conclusions to the readers or discuss them together.
Maintaining the playability of the game allows everyone to value the working game over the comprehensive design.
Damage Meter
During the development of Midtown Madness, we needed to find a way to communicate the level of damage to the player’s car. This was important because if a car accumulated 100 percent damage, it stalled, and the player lost the race.
We first drew a meter that grew as damage increased. During play-testing, we found that half the players quickly and intuitively understood that a growing meter indicated growing damage, but the other half did not.
We then switched to a health meter that shrank as the car became more damaged and “lost its health.” Once again, play-testing showed that half the testers understood it immediately, while the other half did not.
So, we went back to the drawing board. How could we communicate the car’s damage level in a way that every player would immediately understand? Someone suggested we communicate a wider range of damage on the car’s body. It was a great idea. We already had some body-damage effects, but it wasn’t enough. So, we added wobbling tires, broken side mirrors, smashed windows, and smoke that darkened as the car approached destruction. At one point, we even added flames, but the vehicle licensees vetoed that effect!
At the next play-test session, all the players understood the damage level of their cars!
The Future of QA
As the game industry continues to improve Agile practices, the role of QA continues to transform. As testing becomes more white-box driven, testers require more knowledge about development disciplines, and testing specialties will grow. For example, test engineers in many industries focus on the quality of the code being produced by programmers. These specialists run code-quality scanner tools3 and measure unit test coverage to ensure that coding practices are being held to a high standard. Is this a testing role or programming role? It’s a bit of both, and it represents a typical evolution and blurring of roles in an Agile environment.
3. Such as PC-Lint
The rising importance, skill level, and value of testers on an Agile game project will improve the role and make it a truly desirable career.
Agile Production
Producers are the most open to adopting Agile but usually with some trepidation. They are the first to foresee its benefits, but they are also concerned about the implications to their role on an Agile game project. The main concern is that as teams become self-organizing and self-managing, then there will be no need for producers. In reality, producers are often the engines of ongoing Agile adoption. As with the discipline lead roles, the production role is divested of many of the mundane tasks with Agile, such as tracking hourly estimates and solving the myriads of small problems that cross-discipline teams handle themselves. Instead, Agile producers expand their focus on the big picture while helping teams achieve continual improvement in how they create games.
This section identifies the changes in responsibility for traditional producers on a Scrum project and how they can assume the Scrum Master, Product Owner, or Product Owner Support role.
The Role of a Producer on an Agile Project
Producers are often seen as the person who makes sure everything gets done. This is not the case on a Scrum project. As Scrum teams take responsibility for what is done every Sprint, they relieve the producer of some daily project management tasks:
Building and maintaining detailed schedules: Scrum teams create and estimate their own tasks.
Tracking daily tasks for each member of the development staff: This is managed in the Daily Scrums by the team.
Managing dependencies: Cross-discipline teams manage dependencies daily and in Sprint Planning by selecting goals and team members who avoid external dependencies.
Instead, the producer should focus on the larger project management challenges:
Tracking external dependencies: Will the outsourced cinematics be delivered on time?
Publisher collaboration: Are the stakeholders playing the game if they can’t attend a review?
Outsourcing support: What does the outsourced level art team need to complete its work?
First-party (platform) support: When are we scheduled for acceptance testing, and what are the requirements?
Risk management: What are the external schedule and resource risks for obtaining licensee approval or assets?
Critical chain management: What are the resource schedule needs for the production assets?
Most of these responsibilities are best described in a number of good game production books (see the “Additional Reading” section at the end of the chapter).
Producer as Scrum Master
Many studios adopting Agile struggle with identifying people to best fill the Scrum Master role. It’s often assumed that the producer is natural for the role, and many times this is true. There are benefits and potential drawbacks with a producer taking on the Scrum Master role.
A benefit seen with producers in the Scrum Master role is the ability to communicate equally well with all the disciplines. This is essential for a cross-discipline team. This allows him to remain unbiased toward issues or unconflicted about what he needs to do. For example, programmers who become Scrum Masters often see issues through a filter that says most problems are best solved with code. This leaves the other disciplines feeling less supported.
Another benefit is that producers usually have fewer tasks to commit to every Sprint. This prevents them from having to choose between completing critical tasks and supporting the team. Depending on the team, the Scrum Master role can take anywhere from 33 percent to 100 percent of a person’s time.
A common drawback with a producer taking the Scrum Master role derives from his past duties on more preplanned projects. These duties required producers to lead individuals toward task completion. However, Scrum teams manage their own tasks to achieve a Sprint Goal as a team. The Scrum Master does not interfere with the team by creating, estimating, assigning, and tracking tasks for them. Unfortunately, some producers assume, out of habit, that the Scrum Master role continues to focus on tasks. This prevents teams from achieving all the benefits associated with self-management. The transformation of a producer-as-task-manager to a team coach is a challenging transformation, but one that results in a far more interesting and satisfying role. Chapter 19, “Coaching Teams for Greatness,” explores this coaching role.
QA as Scrum Master
We often found good Scrum Masters by recruiting and training them from our pool of testers. My theory is that testers are not biased by any one discipline and aren’t seen coming from a management or authority position by the team.
Producer as Product Owner Support
On large game projects, the demands of a Product Owner’s time has led to hierarchies of Product Owners (see Chapter 11, “Faster Iterations”). Similarly, the long-term demands of production planning, resource requirements, marketing, licensing, and first-party hardware support have led Product Owners to seek production support.
As a result, pairing a Product Owner with a senior producer is a common practice. The Product Owner manages vision while the senior producer attends to many project management details. This support enables the Product Owner to make better decisions based on license, franchise, budget, and schedule limitations.
License and franchise details are usually managed by the Product Owner, but there are details that need to be handled on a day-by-day basis. For example, providing sample assets or seeking approval on the use of brands or likenesses can be very time-consuming.
The most important schedule and budget limitations are those that define the production requirements and dates. Pre-production creates cost and schedule debt that is paid off in production. The Product Owner needs to monitor this debt to ensure it doesn’t grow beyond the limits of schedule and cost defined by the stakeholders. For example, before defining the number of characters to produce, the cost for a single shippable character needs to be understood.
A Product Owner and senior producer make an effective pair to create the best game at the right cost and schedule.
Producer as Product Owner
Many successful games have been led by a senior producer who provides project management and vision for the game. As with lead designers who have strong project management skills, a senior producer with a strong vision of what the market wants is rare and invaluable.
When a studio has such a producer, the Product Owner role is ideal for him or her. The role leverages vision and project management strengths like no other (see Chapter 3, “Scrum”).
By Any Other Name
I often visit EA studios where the traditional production role is filled by people who have the title of “Development Director” and where the responsibility of people who have the Producer title is similar to that of Product Owners.
It doesn’t matter so much what the titles are. Identifying responsibilities and discussing how those might best map to the Scrum roles are more useful. Most game teams find that the responsibilities of producers don’t completely disappear when adopting Scrum. There are plenty of things left for them to still do!
The Future of Production
As the industry becomes more Agile, the producer role will include specialized and team roles.
Teams no longer need the ubiquitous “gopher” but will need the support of the following specialists:
Outsourcing/insourcing
Licensing
Franchise management
Production planning and support
Technical production (including first-party communication)
Producers can successfully assume the Scrum Master and Product Owner roles as well. Production will not disappear from Agile studios but flourish.
What Good Looks Like
Roles don’t change overnight, nor should they. Change requires some stumbling and exploration that takes a while to adjust to. QA department managers don’t easily surrender their testers to work on teams, and producers don’t usually hand off task tracking to teams immediately. This is a transformation that takes time.
The same willingness to surrender applies to producers. Control through task management and detailed process rules rarely leads to optimal results. At best, one gets compliance and not engagement
QA that transforms to true “Quality Assurance” and producers who have the courage to relinquish the illusion of control and transform from task manager to coach are at the heart of successful Agile teams.
Summary
Agile QA and production will continue to draw people who have these primary strengths:
A deep passion for video games
Great organizational skills
Great communication skills
These strengths are in demand in every industry, and the game industry has to compete for them like any other. By leveraging them, Agile organizations reap major benefits from QA and production, while testers and producers receive greater challenges in which to rise.
Additional Reading
Chandler, H. 2008. The Game Production Handbook, Second Edition. Sudbury, MA: Jones and Bartlett Publishers.
Crispin, L., and J. Gregory. 2009. Agile Testing: A Practical Guide for Testers and Agile Teams. Boston, MA: Addison-Wesley.
Hight, J., and J. Novak. 2007. Game Development Essentials: Game Project Management. Clifton Park, NY: Delmar Cengage Learning.
Irish, D. 2005. The Game Producer’s Handbook. Boston: Course Technology PTR.
Isbister, K., and N. Schaffer. 2008. Game Usability. San Diego: Morgan Kaufmann.
Laramee, F. D. 2003. Secrets of the Game Business. Boston: Charles River Media.