TESTING
In This Chapter
• Testing Schedule
• Test Plan
• Testing Pipeline
• Testing Cycle
• External Testing
22.1 I
NTRODUCTION
T
o many people outside of the game industry, testing seems like a glam-
orous job. After all, you get to play games all day! However, if you talk
to anyone who has tested games for a living, you know that it is any-
thing but glamorous. In reality, testing is an extremely stressful and difficult job.
Most testers spend at least five to eight months testing the same game day in
and day out, looking for defects, confirming bug fixes, and play testing missions.
This becomes pretty tedious after a few weeks, no matter how fun the game is.
Oftentimes, because testers are looking for specific issues with the game, they
don’t even have a chance to actually just play and enjoy the game.
Another thing that adds to the stress of testing is that many game devel-
opment schedules rarely allot ample time to test everything thoroughly, which
means the testers are often working massive overtime (late nights, weekends,
and holidays) to get the game tested and ready for code release. One reason this
happens is because testing is the last thing to happen in the production cycle.
So if things are running off schedule for art, engineering, or design, these delays
Chapter 22
362 THE GAME PRODUCTION HANDBOOK, 2/E
are amplified by the time testing begins. Testing time is often the first thing cut
when extra production time is needed.
The producer must work closely with the lead QA analyst to alleviate as
many of the testing problems as possible. The lead QA analyst is responsible for
testing the game, closing bugs, and determining whether a game is ready to be
code released. Involve the QA analyst in pre-production so he can comment on
any features that will propose testing challenges. For example, if there are plans
to include 200 options in the character creation system, the QA analyst can com-
ment on how much testing time it will take to test each option and the different
combinations. The testing time alone is probably reason enough to drastically
limit the options for this feature. Things like this will help create a tighter loop
between the development team and the testing team, which will hopefully trans-
late into more manageable testing schedules.
22.2 TESTING SCHEDULE
Since testing time is often cut short to accommodate other schedule slips, create
a solid testing schedule during pre-production. This ensures that everyone on the
team has a clear understanding of the testing schedule and what the expectations
are. If the team understands how their delays negatively impact testing, they will
be more conscientious about meeting their deadlines in a timely fashion. Refer
to Chapter 16, “Game Plan,” for detailed information on creating a schedule.
Build the testing schedule directly into the production schedule and show
the testing dependencies so that delays affecting the test cycle can be immedi-
ately seen and mitigated. Also, add in testing time for each major milestone of the
game so the testing department can spend a few days with a single build to evalu-
ate the game’s progress against the milestone deliverables. Refer to Chapter 15,
“Game Requirements,” for details on milestone deliverables.
Other things to include in the testing schedule are as follows:
Play testing: During production, make sure to schedule time for QA to play
test the game and offer feedback to the developers. Ideally, conduct these
play tests with people who haven’t already spent months playing the game,
so the feedback is based mainly on the fun factor of the game.
Demo: Marketing will want a demo, and it will need to be tested. If the
demo is already in the schedule, everyone will be prepared to fulfill this re-
quest when marketing makes it.
Marketing builds: If marketing is sending development builds out during
production, schedule time for the testing department to check them be-
fore they leave the building. Even though journalists expect to see bugs and
TESTING 363
unfinished work in these builds, there might be critical errors uncovered in
testing that must be added to the build notes.
Code release candidates: After beta, the development team’s main goal is
to get the bugs fixed as quickly as possible and create a suitable code release
candidate. Schedule a few weeks at the end of the testing cycle for the QA
department to thoroughly check each code release candidate against the test
plan. If things go well, the first code release candidate will pass with flying
colors, but more than likely, several code release candidates will need to be
submitted and tested.
During pre-production, the testing team is mainly used as a resource for
play testing the prototype and offering feedback on proposed features. At this
point in the testing schedule, the QA analyst can be on the project part-time, as
long as he has can provide feedback on all the deliverables being generated at
this time. If there are prototypes or playable builds to check, he can arrange to
have a few testers available for a few days at time to test these deliverables dur-
ing pre-production.
Production is where the bulk of testing takes place. The QA analyst will be
on the project full-time after production starts—working on the test plan, testing
gameplay features, and working with leads on managing the production pipeline.
A few testers will be needed for a few weeks here and there before alpha—a
team of full time testers is not necessarily needed until the game reaches alpha.
However, if the game is very large and complex, there might be plenty to test
before the game reaches alpha. Remember that the sooner something can be
tested, the sooner bugs are uncovered and fixed. In some cases, a difficult bug
to fix found later in the development cycle was an easy bug to fix earlier in the
cycle.
At alpha, the QA analyst will put together a group of full-time testers who
can test the features as they are implemented. At this point, it is likely the testing
team will not have reached full capacity, as the full game is not completed. After
code freeze happens, about three to four months before the game is scheduled
to code release, expect to have a full group of testers looking at the game until it
is finished. If the game is especially content heavy, the full group of testers might
begin looking at the game before code freeze.
22.3 TEST PLANS
The QA department follows a test plan in order to thoroughly check all areas of
the game. The QA analyst will base an initial test plan on the design documenta-
tion and then will work with the team to keep it updated during the production
process. As discussed earlier in the book, it is useful to have the QA analyst on the
364 THE GAME PRODUCTION HANDBOOK, 2/E
team as soon as possible so they can start flagging potential testing issues and start
writing the test plan. Depending on the scope of the game, the test plan could
be hundreds of pages. The test plan is usually presented in some type of pass/fail
format or a checklist. For example, a checklist may include a list of playable char-
acters, and the tester needs to play the game and check off which playable char-
acters are in the game. In a pass/fail format, the tester may need to check each
button on the UI screen and note if it passes (meaning it functions as expected),
or fails (meaning it does not work at all, or does not function as expected).
It is critical for the test plan to be updated to reflect the latest changes to the
game and the design documentation. Valuable production time can be lost if the
QA department is using an outdated version of the test plan to check a build. For
example, the lead artist may have approved cutting some of the playable charac-
ters in the game, but didn’t inform the QA department or update the character
asset lists. A QA tester starts checking the list of characters against his test plan
and finds that several characters are missing. This gets logged into the database
as a bug, and now needs to go through the testing process in order to verify that
the game is correct, and it is the test plan that needs to be updated.
Figure 22.1 is an example of a pass/fail test plan. In this example, the test
plan is written in Microsoft Excel in order to best organize the information to
FIGURE 22.1 Sample pass/fail test plan.
O
N
T
H
E
C
D
TESTING 365
FIGURE 22.2 Sample checklist test plan.
O
N
T
H
E
C
D
be checked. As testers play through level 1, they check the things detailed in the
test plan and mark if items pass or fail. If an item fails, the tester will include
some notes as to why. If testers are unable to check a specific item, they will
mark an item as “CNT” which stands for “Can Not Test” and add notes as to why.
Note that each level is listed on a separate worksheet, and each level will have
unique objectives and enemies listed in the test plan. Figure 22.2 is an example
of a checklist test plan. In this example, there are three playable characters and
they can use any of the weapons listed in the sheet. Each weapon has up to three
levels, with a level 1 weapon being less powerful than a level 2 weapon, and so
on. The tester selects one of the playable characters and a weapon, and then
cycles through the weapons levels while playing the game. They check off each
combination as it is completed. If there are any problems, the tester will enter a
note about it, and the item will remain unchecked.
The test will be comprehensive, but the QA department will not test every build
they get against the test plan. Instead they will focus on rotating through certain
section of the test plan on each build they get. If they receive specific instructions
to test through a portion of the game, they will consult the test plan for information
on how to best fulfill this testing request. Milestone builds are also fully checked
against the test plan, as this is a good way to benchmark the progress of the game.
As the game gets closer to shipping, it becomes more important to check
the test plan thoroughly. Once there is a gold master candidate for the game, it
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.