366 THE GAME PRODUCTION HANDBOOK, 2/E
is critical to schedule enough testing time so that the QA department can run
through the entire test plan from start to finish on the gold master candidate.
The testers may discover some minor errors that need to be addressed before
the game ships.
22.4 TESTING PIPELINE
Before testing begins, the testers and the development team need to determine
the pipeline for tracking and reporting bugs. If this is not established, there will
be confusion about how the testers are reporting bugs and which bugs are most
critical for the development team to address first. In addition, everyone needs to
understand how to access and use the bug tracking database.
Bug Tracking Database
In order to efficiently track bugs, a centralized bug tracking database is critical.
Don’t rely on emails as a reliable form of bug-tracking. Instead, set up a bug-
tracking database, such as Seapine’s TestTrack or Bugzilla. Both of these pro-
grams offer robust bug-tracking functionality for writing and closing bugs.
After the database is set up, the QA analyst can conduct a tutorial to train the
team on using it. The team must understand how to use the database correctly
so they can enter bugs, comment bugs, change the status on bugs, and basically
understand what they must do to address their bugs in the database. During the
tutorial, the QA analyst can also discuss bug definitions, so that everyone has
a common understanding of the differences between crash bugs, critical bugs,
minor bugs, and feature requests.
Bug Definitions
When bugs are added to the database, make sure the correct bug definitions are
used so that the bugs can be fixed in the most efficient order. For instance, crash
bugs should be addressed well before any minor bugs or feature requests. If the
bug is not properly defined in the database, crash bugs might not be addressed for
a while and will ultimately become more difficult to fix as production continues.
Additionally, if feature requests are defined incorrectly as bugs, feature creep will
sneak up on you before you know it. Common bug definitions are as follows:
Crash bug: A crash bug is extremely serious as it prevents the player from
progressing in the game. Crash bugs can freeze the game, or in the worst
cases kick the player out of the game and display an error message. The “blue
screen of death” sometimes seen in Microsoft Windows is a crash bug.
TESTING 367
Critical bug: A critical bug is a major functionality problem with the game,
but it does not prevent the player from progressing in the game. A critical
bug is a level missing all of is textures, or a major gameplay feature not func-
tioning as designed.
Minor bug: A minor bug is one that is noticeable by the player but does
not detract greatly from the overall game experience. Stretched textures and
typos can be considered minor bugs.
Feature request: A feature request is not a bug, so be sure everyone en-
tering bugs in the database clearly understands the difference. A feature
request is additional functionality that would be nice to add but that is not
part of the defined feature set. For example, someone might request an
option to turn the in-game Heads-up-display (HUD) off and on, but if this
feature was not an original part of the design scope, it is considered a fea-
ture request and not a bug. If the user is supposed to have the ability to
toggle the HUD on and off, and this is not working in the game, then this
is a bug.
When writing the bug, there is usually a section for including information
on the type of bug it is. Please refer to the “Writing Bugs” section later in this
chapter for detailed information on writing bugs.
22.5 TESTING CYCLE
The testing cycle for a build begins when the development team officially sub-
mits a build for testing. As discussed in Chapter 19, “Making Builds,” even
though builds will be available on a daily basis around alpha, it is not useful for
QA to test each every build, as they would never make it through the entire
game before a new build was ready. Instead, if the build is stable, the QA de-
partment can spend a few days or a week with a single build and test as much
as possible.
As development progresses and the game becomes more robust, the QA
analyst will test different sections of the game on different builds. For example,
when a level artist checks in a new level, the testers will do a pass-through on the
geometry and textures and submit any bugs to the database. This level will not
be tested again until the artist has fixed all the bugs and resubmitted the level for
testing, which could be several weeks. Meanwhile, the testers will concentrate
on testing other levels and features in subsequent builds as they wait for things
to be fixed. Work with the QA analyst to schedule certain sections of the game
for testing. If it is indicated in the schedule when certain parts of the game are
368 THE GAME PRODUCTION HANDBOOK, 2/E
supposed to be ready for testing, the development team is better able to plan
their work so they can accommodate the testing schedule.
After the build is in testing, the testing cycle is fairly straightforward. The
testers will run through the test plan, find bugs, and enter them in the database.
When the bugs are in the database, they are assigned to the appropriate person
for fixing. This person fixes the bug and resubmits his work for verification in a
future build. The tester will then check the fix in the build and indicate that is it
is ready to be closed out of the database.
Writing Bugs
Once production and testing are in full swing, the central bug tracking database
is the most valuable resource for tracking the progress of the game. Members of
the development team should be required to input any bugs they encounter in
the database, along with any feature requests, or feedback that requires a change
in the game. While a feature request and feedback are not bugs, it is good to
include them in the database so they can be tracked, addressed, and verified in
the game. If a feature request is not approved for inclusion in the game, it can
be tagged as a feature request and addressed for any future patches or sequels
to the game. During a code freeze, engineers usually don’t make any changes
to the code unless it is specifically recorded in the bug database and assigned to
someone to be fixed.
Because the team will have so many people writing and entering bugs in the
database, it is imperative to establish a standardized process for writing bugs and
training everyone on it. The bugs need to contain the pertinent information so
that the development team can easily figure out what the bug is, find it in the
game, and fix it. Most bug-tracking databases have a standard set of information
fields to be filled in when writing a bug. These fields include:
Version: This is the version of the build where the bug was found. The ver-
sion is useful for tracking the progress of the bug from the time it is found, to
when it is fixed, and then when it is verified. If the bug pops up again later on
in the cycle, the version history will be useful information for tracking down
the cause of the bug.
Category: This indicates if it is an art, design, or engineering bug. Usually
this is fairly easy to figure out, but if in doubt the tester will make a best
guess as to the category. When the appropriate lead takes a look at the bug,
they may re-categorize it.
Component: A sub-category of “category.” This offers more details on what
behaviors the bug is exhibiting. For example, sub-categories in “engineer-
ing” might be networking, AI, UI, physics, load/save, and so on.
TESTING 369
Summary: A quick one-sentence summary of the bug. The team may es-
tablish specific guidelines how to write the summary so that the bugs can be
easily sorted. For example, all art bugs found in mission one need to begin
with “M01—Art.”
Description of bug: The person writing the bug needs to describe what
happened. In some cases, he may want to include information on what he ex-
pected to happen and what actually happened. This allows the team to easily
identify any bugs that are not working as designed, or in some cases, features
that are working as designed, but for some reason are perceived as bugs.
Severity: This indicates whether a bug is a crash, critical, minor, or a feature
request. Severity One bugs are addressed first, as they are usually crash bugs
that prevent the player from progressing in or playing the game. Critical bugs
are classified as Severity Two and usually indicate bugs that block gameplay.
Minor bugs are usually considered to be Severity 3, while a feature request
is a Severity four.
Priority: This category is another way to classify bugs and indicates which
ones are highest priorities. For example, you may have three crash bugs all
designated as Severity One. However, one of the crash bugs happens at the
beginning of the game, while the other two happen at the end. The producer
or lead may designate the crash bug that happens at the beginning of the
game as Priority One and the others as Priority Two. This helps the team to
make decisions on which bugs need to be addressed first.
Steps to reproduce: This provides a step-by-step description of how to
reproduce the bug (if it is reproducible). If the bug is not reproducible, the
testers should note down in a chronological order what he was doing when
the bug occurred. People should make it a practice to reproduce the bug
whenever possible.
Screenshots: Including a screenshot of what was happening on screen at
the time the bug occurred is very helpful in pinpointing the location and
cause of the bug.
Crash log files: The engineer can create a debug executable that will gen-
erate a log file each time the game crashes. The log file will note which line
of code the crash occurred in, so the engineer has a good starting point for
tracking down a bug.
Assigning and Closing Bugs
Assigning bugs is a large part of the testing cycle, because the bug has to get to
the right person in order to be fixed and verified. The process for assigning bugs
should be clearly defined and presented to the team during the testing tutorial
370 THE GAME PRODUCTION HANDBOOK, 2/E
conducted by the QA analyst. The goal of assigning bugs is to get bugs addressed
as quickly as possible.
A simple process for assigning bugs involves the tester, the QA analyst, and
the lead:
1. The tester finds a bug in the game, writes it up, and submits it to the
database.
2. The bug is automatically assigned to the QA analyst who will check the
bug to ensure that it is indeed a bug and not a duplicate. He will also
check the information in the bug to make sure that the problem is clearly
written.
3. The QA analyst will assign the bug to the appropriate lead. The Art lead
receives all the art bugs and so on.
4. The lead will review the bug, verify that it is assigned to the appropriate
discipline, and assign it to the appropriate person on the team. In some
cases a bug appears to be one type (such as art) and is really another
type (such as engineering), which is why it is helpful for the lead to re-
view the bugs. If the bug is a crash bug, the lead might request that the
person address the bug right away so it can be fixed for the next build.
5. After someone is assigned to fix the bug, he will implement a fix and
assign it back to the QA analyst for verification. If the bug can’t be fixed
for one reason or another, the person should add a comment to the bug,
explaining why, and assign it back to his lead for verification.
6. After the analyst is assigned a bug for verification, he will assign it to a
tester to check in the next build to make sure that it is fixed. If the bug is
not fixed, the tester will add a comment to the bug, and the process will
begin again. If the bug is fixed, it is closed by the analyst.
Fixing every single bug in the game is not possible, especially as you get
closer to code release. Most developers have a list of “will not fix” bugs that will
remain in the game when it ships. Some of these bugs may be addressed later
with an update or patch, but most will not. Bugs categorized as “will not fix”
are minor bugs that will not demonstrably impact the player’s game experience.
The bugs may be cosmetic—there is stretched texture or a visible seam in a 3D
model. Other bugs may be related to gameplay, but are prioritized as low risk
and not worth jeopardizing the ship date to fix. Each developer has different
standards for designating a bug as “will not fix,” and will likely need to get ap-
proval from senior management for anything on this list.
Checking TCRs
As discussed in Chapter 5, “Developer and Publisher Relationships,” each con-
sole manufacturer has a predefined set of technical requirements to which each
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.