3

How to Run Successful Specification Reviews

“He who knows only his own side of the case knows little of that.”

- John Stuart Mill, On Liberty

In the previous chapter, you learned how to develop an excellent feature specification from a short description given by a product owner. However, even with all those details added, your test plan is incomplete. So far, the only inputs into the test plan have been that initial draft and your exploratory testing and ideas. You also have to hear from the developers to understand their choices, the configuration options they added, and the areas they are worried about. From the product owner, you need clarification on any unspecified user-facing questions. The specification review meeting is the place to gather that information.

This chapter will show you how to run a successful review meeting. As with the functional specification, this may be carried out by someone else, but if it isn’t, it’s up to you, as a tester, to ensure this is in place. You need a specification to test from, and you need everyone to agree to it. This chapter describes how to get that done.

We’ll cover the following topics in this chapter:

• Why do we need a specification review?
• Advantages, disadvantages, and alternatives of specification review meetings
• Inviting the right people
• Scheduling the meeting
• Running the meeting
• Fixing incorrect requirements
• Opening the black box
• Setting a good tone for the meeting
• Prioritizing requirement testing
• Review meeting checklist

We’ll describe how to structure the meeting and set the tone, and what questions to ask to draw out important points. Finally, you’ll learn how to prioritize the testing and run through a checklist to ensure your review is complete.

Why do we need a specification review?

The goal of the specification review meeting is to agree on all the requirements for this feature. By the end of the meeting, you should know exactly what the feature specification should say, even if it’s not fully written up yet. Armed with that agreement, the test and development teams know exactly what they should deliver.

The specification review requires getting the right people to the meeting and giving them sufficient time to prepare. They may or may not take advantage of that time, but it’s important to offer it.

Of course, you can have many other discussions during the feature development aside from the review meeting. Keep communicating before and after review; this meeting isn’t a replacement for that. However, the review meeting serves as a final, systematic check that the specification is complete and ready to go.

In my experience, developers are only too happy to have a tester checking that their feature has been implemented well. No subject is off limits at this stage of the project because you are still planning out the testing you will carry out. After the feature review, finding gaps in the test plan becomes less positive since there is always the question of why these issues weren’t discovered earlier. A successful review meeting avoids those surprises later.

First, we will consider the advantages and disadvantages of a review meeting and some alternatives.

Advantages, disadvantages, and alternatives of specification review meetings

The key advantages of the specification review stem from the communication between the key stakeholders. Its main disadvantage, as with writing the specification itself, is the time it takes. It’s your job, as a tester, to convince the project team that the investment of time is worth making. Here are its advantages and disadvantages:

Advantages	Disadvantages
Gets all the key stakeholders together	Takes the time of senior team members
Structured discussions are made based on the test plan	Difficult to set the right tone
Chance for rapid discussion and decisions	Requires a detailed specification to be useful
Chance to uncover problematic scenarios

Table 3.1 – Advantages and disadvantages of specification review meetings

Advantages of specification review meetings

Getting all the key stakeholders together for a detailed discussion of a feature is a massive boost to a project. This should be relatively late in the development cycle when the first implementation is already running, and you have performed exploratory testing. You’ve had a chance to get your hands on the feature, and you’ve drafted the specification. Now, you can examine the feature in detail and check for any gaps or omissions in its design.

The specification provides a detailed structure for examining all the minutiae of the feature. Having written the specification, you should be well acquainted with the feature’s workings but you’ll still have questions about different areas. This is your chance to ask those questions and solicit for any unknown unknowns – the questions you didn’t know you should have asked.

Having asked those questions, you can rapidly uncover any problems. Sometimes, there are bugs in the specification or scenarios that haven’t been considered yet. For other cases, you can quickly discuss and decide how the feature should behave. A well-run review provides detailed, swift, and decisive answers.

Disadvantages of specification review meetings

The disadvantage of the test plan review is that it requires time and thought from senior members of the project, who tend to be very busy. I believe this meeting is worth their time and will more than repay that investment, but some team members might need persuading.

Another difficulty is setting the right tone. Often, testers come into a project relatively late and have spent far less time thinking about a feature than the product owner and developer, who might have been working on it for days or weeks. You’ll have many questions to ask, which can be embarrassing. You need a thick skin and confidence in your abilities to keep questioning until you have the answers you need.

Conversely, some developers might see such questions as a threat, so they need to be asked respectfully. It can be a tricky line to tread, asking for senior developers’ time for the review and then checking their work. Staying polite, positive, and impersonal will help smooth the way.

As well as the investment of time, the specification review meeting requires a specification to review. If you haven’t put in the time to write that document, you can’t get the benefits of the review to draw out further details.

Alternatives to specification review meetings

The ISTQB has a lot to say about meetings: the different kinds you can have and the attendees who should be there. You can review the user interface, the technical implementation, or the functional behavior of a feature. Here, we will only consider their usefulness for reviewing functional specifications because the test plan is based primarily on that. The ISTQB lists four types of review:

• Informal reviews: Informal reviews have no specified agenda or intended outcomes but are unstructured discussions about the products and their workings. Like exploratory testing, their unstructured nature means their usefulness will depend greatly on the individuals taking part, and senior team members are likely to achieve more with them. There is no requirement for informal reviews to be documented, but they can be; photographs of whiteboard diagrams can be used to produce more official documentation later, for instance. These ad hoc discussions about crucial points are an important way to exchange information, especially if you need to know an answer before further testing. They are quick and don’t consume many resources, although their usefulness will vary.
• Walkthroughs: Walkthroughs are more formal. They are led by the author, either of the code, the design, the specification, or whatever is under review. Their structure is variable, and they can be more or less formal, depending on how you would like to run it, but they all have the same goal: to examine a particular project asset. They tend to be open-ended, so you can quickly cover easy, obvious sections and have deeper discussions on any areas of contention or confusion. They help other team members understand more and are a chance for a reviewer to find defects.
• Technical reviews: Technical reviews include multiple different tasks. There will be discussions, some of which involve evaluating alternatives and choosing between them. You may solve technical issues and, most notably for testing, identify defects at the specification level, such as missing or contradictory requirements. While the specification document provides a structure to the review, the format is very flexible and can be whatever you need.
• Inspections: The final form of review listed by the ISTQB is an inspection. This is a formal acceptance of a change and might be used for code reviews, for instance, before developers merge changes into a product. Peers often conduct inspections, and they include a documented discussion and a pass-or-fail outcome. It’s mainly used to find defects as a check alongside the testing.

The review I propose here is what the ISTQB calls a technical review. It’s a structured discussion of a particular document, which in our case is the functional specification, which acts as a checklist to work through. It requires a draft of that document as an input and will produce a reviewed version as an output. Reviewers need to attend, both those working on the feature and possibly senior members of the team as well. For the test manager, this is the most important meeting to attend. You get to see a complete description of the feature, participate in discussions around the problematic areas, and see the testing that needs to be done.

The different strengths and weaknesses of these reviews mean they are beneficial at different times. Informal reviews help the team understand the feature and answer specific questions, and walk-throughs can be an extended version of that, especially to introduce newcomers to a planned feature and give an overview. Inspections are especially valuable for code reviews between developers. But for testers, I recommend a technical review of the feature specification as the key meeting. It provides the structure and flexibility to thoroughly examine a new feature, so long as the right members of the team are present. The following section considers the necessary guest list for a successful specification review.

Inviting the right people

The most important part of the specification review is inviting the right people to the meeting. If you get that wrong, nothing else you can do will make the meeting a success. So long as you have the right people and a detailed specification to work from, you are ready to begin. So, who should you invite?

While there are many roles necessary to make the specification review a success, three people are critical: the product owner, the lead developer, and the lead tester. Optionally, the senior developer, user experience team, and project manager can also attend:

Figure 3.1 – Attendees for specification reviews

The ISTQB suggests six different roles for specification reviews. Personally, as a test team leader, I was usually all of those, performing multiple different roles simultaneously, so I don’t consider them necessary. However, it’s worth describing the meeting roles, especially if they will fall to you. As described previously, ideally, you will be presented with a detailed, reviewed specification to base your testing on. However, since that may not be the case, you need to know how to do these tasks and do them well:

• Leader (arranging the review meeting): Being the meeting leader means ensuring that the specification review happens by finding availability and sending out invitations to all the stakeholders.
• Manager (allocating resources to testing): The manager performs the vital task of assigning the resources. If the project doesn’t have the time or equipment to succeed, there’s no point in planning out the testing. Since this book doesn’t cover project management or prioritizing tasks, I will assume you have the resources and can concentrate on planning out the tests.
• Author (you, the person who wrote the specification under review): The author of the specification should attend to present their work. Again, here, I assume that is you. Even if you didn’t write the feature specification, as the tester, you need to be thoroughly familiar with everything it says. Writing it yourself is an excellent way to ensure that.
• Scribe/recorder (documenting the outcome of the meeting): The scribe plays a vital role. If you don’t write down the results of the discussions, then points will be missed when you look back. Ideally, the scribe should present their notes during the meeting so that everyone can see what notes they are writing. That way, the minutes aren’t just the scribe’s recollection of discussions and decisions; everyone can see what they agreed on. If you can delegate or find a volunteer for this, I highly recommend it. Let someone else write notes so that you can concentrate on facilitating the discussion.
• Facilitator/mediator (resolving conflicts): The facilitator plays the important role of drawing out questions, shutting down side-discussions, summarizing conclusions, and keeping the meeting moving forward to ideally finish on time. This is a skill in itself and worthy of its own book, so here, I will just note that it is necessary. Specification review meetings have a great structure – you step through the requirements, line by line, so that they have a clear start and end. The review is also documented by default because if no one says anything, then the specification is correct as it stands. Sometimes, the challenge is to make people speak up; other times, you’ll need to take a discussion offline to make progress elsewhere. This will be your main task during the meeting if no one takes on that role.
• Reviewers (providing technical input).

All these roles provide the structure of the meeting. The content will come from the reviewers. The ISTQB notes that there need to be reviewers, but not who they should be, so that role requires additional detail.

There should be a minimum of three people in the specification review:

• The product owner, who describes what the customer requires and makes any customer-facing decisions
• The lead developer, who describes the implementation, including any limitations or special cases that need to be tested
• The lead tester, to ask about error cases, feature interactions, and feature details

The product owner has to be present to advocate for the customer and give the underlying rationale for this change. They should approve any decisions with customer-visible effects. The lead developer can describe the implementation and any scenarios for which they had to write code to ensure the specification covers them. They can also explain how they handled error cases or the current behavior of the code in different situations. And finally, the lead tester must be present. You can ask questions about the feature and present the current specification you are planning to test.

In addition to those three, several other people may be necessary:

• The project manager, to prioritize tasks and decide the scope of changes
• The user experience team, to propose any changes that are needed to the user interface
• The senior developer, who will oversee the development of this feature
• The senior tester, who will oversee the testing of this feature

Discussions in the specification review will often cover features you would like to add but haven’t had time for yet. The product owner will decide what is necessary to ship this feature, and a project manager may support them in planning out that work and when it can be delivered. The project manager can also guide the project if it looks like it will take too much time or too many resources.

The user experience team should be present for any feature with significant user-visible changes. Those can be the most contentious areas, so the UX team should attend to provide their opinions and approve any changes you need to make.

Finally, there should be senior developers and testers present to make sure the junior members of the team are performing well. This meeting is a key handover: it describes the detailed behavior of the feature, how it has been implemented, and how it will be tested. If a junior engineer has worked on this feature, then plan to invite the senior engineer they have been working with, to help answer any questions or make decisions. A software architect should also be involved if the scope of the work is large. For any managers who haven’t been closely involved in this feature, and only come to one meeting to learn details, it should be this one.

While it’s great to have more opinions, aim to keep meetings as small as possible to avoid wasting people’s time and encourage questions and discussions. Having a large group can discourage people from speaking up, but always ensure the key people are present. Otherwise, you will just have to arrange another meeting to get their input too. The trick is to require just one meeting, and the best way to schedule that is described in the next section.

Scheduling the meeting

You should send around the feature specification a few days before the meeting to give everyone a chance to review it and submit comments in advance. If there is a critical misunderstanding – an area of functionality not covered or described incorrectly – it can be flagged up without wasting people’s time in the meeting. It works well to have both a written and spoken review. The reviewers can take their time and think through their comments, and then you can rapidly discuss them and reach decisions.

You may need to have multiple test plan reviews. If you have planned out the feature before it has been implemented, there should be a review before implementation begins. After it has been implemented, there should also be a review to catch any necessary changes. Often, not all functionality will be delivered in the first version, so this is the chance to describe the exact behavior in this version and what will be added in the future.

For large features, it may be necessary to break the review up. These meetings require concentration and discussion, but it can be easy for reviewers to switch off, fail to pay attention to requirements, and fail to raise objections. For this reason, you should aim to keep specification reviews to under an hour. If a specification needs more than that, split it into different meetings. These should be close together so that reviewers remember the detail of what they discussed, but with breaks so that everyone is refreshed. This helps them listen carefully and think through the consequences of every requirement statement.

While you might schedule the meeting for an hour, everyone will appreciate it being shorter than that! Keeping the meeting on track and moving forward successfully is the skill of running the meeting well, as described in the next section.

Running the meeting

The meeting format is simple enough – step through the feature specification, read each requirement to see if there are any disagreements, and sign it off. Was it the behavior the product owner wants? Has the developer implemented it (or have they agreed to implement it)? This is a form of static testing – testing without running the code. This lets team members step through design documents, technical specifications, or, in this case, functional specifications to look for errors or issues. This contrasts with dynamic testing, where you run the code to see how it behaves in practice. Only this chapter, Chapter 2, Writing Great Feature Specifications, and parts of Chapter 6, White-Box Functional Testing cover static testing; the remainder of this book focuses on different types of dynamic testing.

With a large specification that you are already familiar with, you may find it tempting to breeze through the requirements quickly. Don’t. Take your time, read each requirement slowly, and make sure that everyone has heard and understood them. While you have given everyone time to prepare and read the specification, be ready for anyone who has turned up without doing their homework. If this is the first time they are seeing these requirements, read them slowly enough that they can keep up.

Remember that you are looking for objections. Questions are good! Disagreements are good. You might hope that your specification is perfect, and you might be eager to get to the end of the meeting, but take your time and solicit as much feedback as possible. In church, when the priest asks if anyone has any objections to marriage, no one wants to find any. In this case, you want objections. You want to have as much discussion as possible as early as possible, to air any worries anyone has, and to work through them. You are actively seeking communication.

The worst thing that can happen in a specification review is that you miss a requirement and test against an incorrect or incomplete specification. This is your chance for people to raise any concerns they have. Maybe there’s a vague worry they haven’t been able to articulate yet. This meeting should be a safe, supportive environment for them to ask their confused question, even if they don’t fully understand what they’re asking. For a detailed explanation, refer to the Setting a good tone for the meeting section.

If you read through the specification and everyone sits in silence, don’t congratulate yourself on having written a perfect document. More likely, they haven’t read it or aren’t listening and adequately considering it. There is always more to add, so if everyone is sitting in silence, ask them direct questions (or say something outlandish!) to ensure they’re paying attention.

Alternatively, your document may generate too much discussion. As with all meetings, you need to ensure you don’t go down rabbit holes. If there is a particularly contentious area, you’ll need to continue the discussion in a separate meeting. Leave it to one side, and continue with the other requirements, which don’t depend on that decision. If you disagree about something fundamental, which impacts the entire feature, you need to get that agreed upon first. However, hopefully, those decisions are made long before the specification review.

For contentious issues, always summarize them at the end of the discussion. It’s possible that some people still won’t be happy with the outcome, but you need to let everyone know what you are planning to do. It’s no good if people come away with different understandings at the end of the meeting. That’s why it’s so useful for the scribe to present the notes they’re taking as they go along, so everyone can see what’s been decided. That is usually simple enough, but it’s vital for controversial decisions.

These conversations are necessary to decide on any behavior you weren’t sure of when writing the feature specification, as described in the next section.

Fixing incorrect requirements

You have to get feedback on any requirements you weren’t sure of. For any load limits you plan to measure and any failure modes that could be handled differently, you have to ask for confirmation explicitly. As a tester, you need an answer to those questions, but you might not be best placed to make those decisions. The user experience team and product owner will have a better idea, so ask them the specific questions you need.

As previously noted, reviewers may object to choosing a limit for system metrics. They may say there’s no point stating that the system can only support 1 million users, for instance, when in all likelihood it could do many times that number successfully, or conversely, it might not get anywhere near that number in reality. While there may be no difference between 1 million and 10 million users for the development team, there is a large difference in what you have to test and support, so you need to agree on that number.

These are likely to be areas the reviewers haven’t considered before, which will generate some discussion. Except for when it affects testing, it’s best to let the subject matter experts decide those points. Your contribution to the process was asking the question and recording the answer. The same applies to learning about feature behavior, as shown in the next section.

Opening the black box

The most valuable part of the specification review is quizzing the developers on the extra work they had to do. Which special cases did they have to write additional code for? Explicitly ask that question because it may not be evident from either the product owner requests or your exploratory testing what work has been necessary.

You can invite the developers to present the architecture to discuss how the different sections work together and how this feature was implemented. This loses the naivety that was helpful during exploratory testing but, at this stage, it’s much more important to be an expert in the workings of the feature.

Understanding the architecture helps explain which variables are independent of each other and can be tested individually and which you should expand out to test the entire matrix of possibilities. It also helps show areas of the test plan that need to be covered in detail and which can be checked with a single test. This will be examined more in Chapter 6, White-Box Functional Testing.

The developers can also let you know about the areas where they have the most concerns. What do they want to see tested? Are there any far-reaching changes they’ve made that they haven’t been able to check? Listen carefully to that and include those concerns in the test plan. Conversely, a developer being overly confident in an area is necessary but insufficient to suggest its quality. Be wary of those areas too.

Paying attention to the confidence of individual team members and the overall tone of the meeting is important enough to need explanation and is covered next.

Setting a good tone for the meeting

So far, this review sounds like it’s being conducted by polite robots, dispassionately discussing the best way to approach a feature. The team members will already have a lot invested in it, and conflicting characters might cause issues, so your specification review meeting should have a friendly and supportive tone. This is vital to drawing out discussions and resolving them successfully. If you are running this meeting, it is your job to set the meeting’s tone.

Why should a technical book about testing have a section on being nice to people? Aren’t SQL injections, database performance, boundary value analysis, and other things more important? While I will cover those areas, testing involves a lot of communication, both to understand features and to report issues. I promise it will only be this once, but, especially during the specification review, personal interactions can be the difference between a successful feature release and a failure. It’s worth explicitly describing the tone you should set in a review meeting.

Real-world example – A failure to communicate

Early in my career, I worked on a feature with a senior developer. He was a big character around the office. Bombastic, you might say. Though I knew vastly less than him, I thought there was a problem in one section of the code he’d worked on, so I pointed out that he might want to rethink that area.

He replied that he was perfectly happy with that section of code, and we left it at that. Later, the code was released, and the behavior I’d flagged was indeed a bug we had to fix urgently. He asked why I hadn’t told him about that problem. I thought I had, but I’d failed to get my point across, and the product had suffered as a result.

Specification reviews challenge the developers and product managers to see whether they have thought through this feature well. Any contradictions or gaps you find are errors they made. As a tester, you’re a professional at pointing out other people’s mistakes. Software development is such a complex field that fixing it is a full-time job and an entire industry. Despite your experience at finding bugs, perhaps the product owner doesn’t take kindly to you rewriting their specification, or the developers are happy with the error handling they’ve already implemented and are disinclined to revisit that code.

All the techniques you use when raising bugs apply here, too: be specific, respectful, and impersonal. If in doubt, phrase concerns as questions rather than statements. “Is this wrong?” rather than “This is wrong.” Press your case if you don’t get a satisfactory answer; keep asking questions until you’re sure you understand. You need the confidence to be able to ask stupid questions, to start at the beginning, but make sure you fully understand the feature and how it works.

Asking questions is the best way to understand a feature. Officially known as the Socratic method, you can query what the behavior should be in different scenarios or how certain aspects were implemented. As a disinterested observer, you can spur the conversation to find the details you need. Arguing for specific points is good too, but the fundamental goal of the review meeting is clarity. How exactly will this feature behave? If you can specify that, then the meeting was a success.

There can also be tensions within the test team. Writing a long, detailed specification means someone has some long, detailed testing in their future. It can help if the person writing the specification is different from the person who will eventually perform the testing. It can be easier to sign up to work if someone else will carry it out. It worked well when I, as the team manager, wrote specifications for my team members to test, though team members could take turns preparing the specifications for one another. The disadvantage is that you need to hand over the task from one team member to another, but that cost can be worth it.

Having drawn out all the issues and documented the changes you need, you can then start to plan out how to approach your testing. That too needs to be reviewed by these stakeholders.

Prioritizing requirement testing

Describing how to plan and track testing tasks is beyond the scope of this book, but the specification review meeting is an excellent opportunity to prioritize the testing. All parts of the feature specification are not equal; some will be well understood and covered by the developer’s testing and used by product owners, while others will be risky and/or rely solely on the test team. As described previously, the developers can indicate which areas they are most worried about. If it’s possible to disable some parts of the feature, the product owners can tell which areas are the most important to test and release first.

Some areas of the test plan are easy to test, such as customer-facing interfaces. Others are backend systems that may require specialist knowledge or areas that need scripting or test tools to exercise. Those areas that will take the longest to set up should ideally be started first. If you have multiple people working on a project, someone can start with those while another team member starts testing the riskiest areas.

With those details in place, you need a final check to make sure everything is ready before the meeting is over.

Review meeting checklist

The goal of the review meeting is to agree on all the requirements in the feature specification so that detailed testing is possible and the development team knows exactly what to deliver. To achieve that, before you let everyone go at the end of the meeting, make sure you have covered all the following points:

• Has the team agreed to all the requirements in the feature specification?
• What questions are outstanding? Who will answer them, and who will check they’ve been answered?
• Have you asked the developers for any code paths they had to add to cover special cases?
• Are there any configuration options that aren’t listed in the specification?
• Has the team agreed to all the cases where you proposed a requirement or behavior? (That includes any requirements you wrote aiming to be precisely wrong rather than vaguely right.)
• Have you prioritized different areas of the specification? Which sections should you test first?

If all that is in place, you are ready to write up your notes, issue one last version of the feature specification with all the changes included, and begin testing in earnest.

Summary

In this chapter, you learned how to run a successful feature specification review. We looked at the advantages and disadvantages of a meeting dedicated to reviewing the specification and some alternatives. No other meeting gives the same coverage and flexibility as a specification review, making it the best way to finalize and agree on feature specifications.

To run a successful review, you need to invite the right people, including the product owner, lead developer, tester, and user experience designer, for any customer-facing features. System architects, senior developers, and testers may also come along, especially if junior engineers have developed this feature. The project manager can also attend to check on scope changes and their effect on timescales. They should be invited with sufficient time to read the specification and prepare. In the meeting, step through each requirement, either agreeing on it or debating what changes it needs. Keep the tone open and friendly to solicit all possible comments and objections.

One key piece of feedback is knowledge of how the feature works internally. We’ll revisit this in Chapter 6, White-Box Functional Testing, but here, you can get the first description of how a feature’s implementation affects its visible function.

Finally, at the end of the review, you can run through a checklist to ensure that everything has been covered and plan out the priorities for which test tasks to approach first.

The functional specification review is such a vital part of the process that it deserves a dedicated meeting and a dedicated discussion. Whenever I have tried to skip this step, either because a feature appeared small and obvious or due to other work impacting the project, I have always regretted it. You can gain some of the same benefits from informal conversations, but there is no substitute for systematically working through every one of the requirements. Recommend that senior members of the team attend this meeting. If they only attend one meeting associated with the feature, make it this one, because it includes all the details of how the feature will work and it forms the basis of future testing.

With the feature specification written and reviewed, you can begin to plan the testing phase. This is the topic of the next chapter and the next section.