Tim sits quietly as Micky the interim CEO spends three hours talking over the Griffin project post-mortem and ripping into the poor results and anyone he felt was responsible. Rushed release, faulty code [crt 20], frustrated and lost customers [crt 27], and below expectation revenues [crt 26] were his key criticisms. The Griffin project manager looked sheet white, ready for a firing squad and his impending demotion.
Micky zeroed in on Gary, the engineering manager. “You need to fix this! We cannot afford these types of mistakes and hope to survive. I want a report next week on how you are going to improve this. Change something, fire someone, or get something out of that new consultant you hired. We cannot do this again!” Micky storms out of the room. Everyone sits quietly.
Gary looks nervously over at Tim. “Tim, I am going to finish talking with the software engineers. Can you talk with QA and see how we let so many issues go out to our customers?”
“Sure Gary,” replies Tim.
Tim walks over to the QA offices, anxious to let the intensity of the prior meeting cool off. Once there, he found the QA leads Anthony and Nir. Anthony is a rather burly guy. Nir is a bit lean, bald, and definitely looks tired and worn out.
“Hi Anthony and Nir. I am working with Gary to help improve his project management process,” states Tim.
“We need it. I heard Micky crashed the engineering review meeting guns blazing,” Nir replies. “I figure the next place people he is looking to blame is QA.”
“So can you give me a little background on your QA process?” Tim asks, not trying to get too caught up in Nir’s soap box.
“Sure. Whatever. I just finished the Griffin project, the one that just exploded. We worked like mad to finish that with the time they gave us. Anthony over here is in the process of developing the base test plan for the Phoenix project.”
“And how are these test plans put together?” inquires Tim.
“We’ve reviewed the initial feature spec, broken out the key areas, and then detailed the various functional tests and test cases to exercise,” Anthony replies. “Once the engineers get further along in their coding, we should be able to do some initial testing. And then at the end of the project, it’s the final testing phase.”
“So you feel that we are in pretty good shape?”
“It’s the tried and true process we’ve been using for a while.”
Nir is tired and overly frustrated and he reflexively interjects, “We had a pretty good plan going into Griffin too! The engineers always kill us in the end! We had set up our test plans, ramped up our testing teams, and had our test systems all ready. We planned a standard two-month test cycle for the final testing phase. Engineering just could not give us stable code that worked. At first we just could not get anything for weeks. Then when we received the early code and it just crashed so there just was not much to test. By the time engineering had something that did not blow up right away, we had a little more than two weeks left. We complained like crazy, but we were talking to a wall. The client had a drop-dead date, so it was not like we could push out the schedule. The engineers were working like crazy and since it took longer than expected, they took all the time out of the QA schedule. So we did very little testing and shipped out an extremely buggy and poor-quality product [crt 20]. So as far as our plan on paper right now, we look fine. But if engineering is late as it usually is, the plan goes out the window. In fact, the last few projects engineering ran late, QA had to scramble with what little time was left over.”
Anthony looks a little ticked. “We can plan out our side; we know how to set up a good QA test plan. I have done it at other companies and I try to do it here. If engineering compresses our schedule, we just do what we can to work around it. It’s the nature of development. If there is a deadline, it’s not like they are going to wait for QA.”
Tim ponders this for a minute. “Let me walk through a few key questions and see if I can better understand and possibly help improve the situation…”
Define a Clear, Concise , and Measurable Goal
“First let’s level set,” Tim states. “What is the overall goal of QA?”
“To catch issues and help ensure the quality of the product,” Anthony rattles off without a second thought.
“Okay, that is good on a high level, but can we be more specific?”
“Tim, do you mean our gating customer release gating criteria?” Nir sighs. “We have a whole 10-page spec if that is what you are looking for. It includes the various areas of testing, the types of testing, systems we test on, number of leads, number of hours, types and severities of issues, allowable number of issues prior to release, and so on….”
“I was more looking for a concise and measurable goal. What would the goal or mission statement be for QA on the Phoenix project?”
Anthony thinks for a second then says, “To ship a product that meets the QA customer release criteria by the November ship deadline.”
“Perfect. It is clear, measurable, and has specific criteria and dates.” Tim opens up an Excel sheet and adds the following line:
Goal: To ship a product that meets the QA customer release criteria by the November ship deadline.
List the Clear Issues and Their Impact
“So guys, what are the key issues
that prevent us from meeting that goal for Phoenix?” inquires Tim.
“Everything; a ton of things based on my experience on Griffin.” Nir states. “Starting with engineering. They are always late. They give us code that is not ready for testing, but if we wait for code that is stable, there would not be any time for QA. No one respects the QA process. Engineering can run as late as it wants and QA has to make up for it.”
Nir and Anthony argue for 15 more minutes and then Tim steps in and summarizes their issues into an Excel spreadsheet.
They talked for half an hour more, but pretty much agreed that these four issues are the main ones.
Define Intermediate Objectives (IOs)
“So Tim, it is nice that we list some of the issues better, but I do not see how this solves our issues,” Nir states.
“Now that we have the issues clearly defined, we need to look at each issue and see if we can come up with something to overcome it. This is known as an
intermediate objective. By design, the person who suggested the issue gets to be the first to suggest a possible way to overcome it.”
“Well, I had the first issue—Engineering provides the software very late and therefore it limits the overall time QA has—it seems rather pointless, because there is not much we can do to control engineering.”
“So Nir, one flying pig suggestion is that engineering deliver code on time.”
“Pretty much, Tim. It would be ideal if they were on time, but yes, based on my experience with Griffin, engineering delivering on time would be a minor miracle and an unrealistic expectation, despite what their managers and schedules claim.”
“Sometime it helps to look at a wild idea and then come back and see if we can find a way to make it more realistic. For example, I’m working with the engineers on their planning. My understanding is that the Phoenix project is made up of multiple modules. So possibly we could plan to complete some of the modules earlier. It is not the same as all of the code being delivered on time, but maybe we could at least get some of it early,” explains Tim.
“That would at least give us a chance to close out some of the testing on those areas and minimize the overall amount that needs to be done at the last minute,” says Nir.
“It sounds promising,” Anthony adds.
“It’s a start, but I do not think it’s nearly enough to address my concern that engineering is always late and in turn it reduces the time QA has to work,” Nir says.
“Okay Nir, let’s try it from a different angle. Let’s assume that Phoenix will run badly and engineering will run late, which means QA will only have a very limited time.”
“Seems more like reality than any assumption in my experience.”
“So what makes sense to do in this situation, Nir?”
“Typically, our test plans go out the window. There is not enough time for a full cycle, so we just test the high risk areas the best we can.”
“Since we are at the beginning of the project, is there a more methodical way we could set up up effective testing in a short time?”
“Maybe we could modularize our test plans better,” Anthony indicates. “So we will have different sections of the test plan for different areas of the code. We can also set up a high-level test pass as well as a more detailed test pass. In this way, if it is a low risk area or we have tested the code before, we might be okay with a quick high-level test, and when we get to a high risk or new area, we would focus on the longer detailed pass.”
“Anthony, are you sure there would be enough time for all the testing we need?” Nir inquires.
“Well, not all areas take the same amount of time, Nir. With a little bit of work, we could give an approximate amount of time it would take to do a high test pass on an area, and based on that how many overall hours we need to do a complete test cycle.”
“So,” Tim summarizes, “Basically, when we get down to the last few weeks, we can look at which areas we are okay doing with a high test pass and which areas we need to do a detailed test pass, add up the time it takes to run all those tests, and balance it against the time we have left.”
“Pretty much,” says Nir.
“Ideally, we would have the time we need, but at least this is more methodical than the two-week scrambles we’ve done in the past.”
“So Anthony and Nir, are we comfortable that setting up a modular test plan should help us address the concern that engineering provides software late?”
“It’s better than where we are at currently,” Anthony chimes in a bit more enthusiastically.
------------
“So the next issue was raised by Nir as well.—The software going into QA is of poor quality.”
“Yeah, this was pretty annoying. We would be delayed weeks trying to get something out of engineering to test. When we finally received something, it would just blow up as soon as we started. So we could not run any real tests, just acknowledge that it did not work. It was of little value to us and gave engineering the chance to say that had given us code to test for weeks.”
“Was it all the code or just code from a few people?” Tim inquires.
“We have a few Engineers that are a little too quick to throw something that is not complete to QA. The other part is when there are tight deadlines and they are running late, the engineers are rushed and so they are quick to move code into QA.”
“Nir, do you have any voice in what the engineers hand over?”
“Well Tim, one idea is that QA could refuse to test code that does not meet our quality standards.”
“That sounds a little rigid Nir,” states Anthony. “We want the engineers to give us early code so we can start working. The early code just needs to be something that does not just crash as soon as we try and start testing.”
“Is there a test that could be done that would be a little less rigid?” inquires Tim.
“I guess we could set up a simple smoke test,” states Nir. “Some basic criteria such as—the code must be able to load in the system without crashing and the features must be exercisable without immediately breaking. We are fine with early and flaky code, but it needs to have some minimal functionality.”
Anthony jumps in, “We also will need to get the engineering managers to sign off on the smoke test, but it should not be too much of a problem.”
“So overall, if the engineering code passes the smoke test, it can be formally accepted by QA. That sounds pretty good,” Tim summarizes.
-------------
“Tim, the next issue was Anthony’s—QA is blamed for the poor software quality. We tell management that we are never given enough time, but they do not understand.”
“With the detailed high-level and detailed-level test plans, I think we can better quantify the time we need to test and the reasons for it. In the past, we just said three weeks sometimes based on a one-page plan. If we have something more comprehensive and flag what areas we are doing a high-level plan, which areas need more detailed testing, I think they will better understand the time and work involved,” Anthony indicates.
“Anthony, how will you handle the last two-week mad dash? The managers state that everything is okay despite issues, then a few months later, when the customer sees the issue, they get all ticked off that QA did not catch it,” Nir states, concerned.
“What if we have a formal signoff sheet? If they want us to skip certain tests, we can have them sign off on a form saying they approved it.”
“So if a customer issue comes up for an area we wanted to test, we will have the documentation they signed off?”
“Pretty much. It should work. But even if they sign off on not testing something, we still should try to do some of the priority tests off the detailed test plan. Sign off or not, it is still important that we feel that a solid test has been done.”
--------------------
“The last one was—Engineering changes the requirements during development without telling QA.”
“This was a pain and just caused a lot of frustration and wasted time. Once the engineers start coding, they do not go back and update all the specs until much later. The specs are simply not their priority. We would like to talk with the engineers, but their time is pretty limited,” Nir states.
“In the past, I’ve informally had coffee with some of the engineering managers and they often fill me in on some of the key changes. The engineers can’t change something significant without their manager’s approval. Maybe we just need to formalize this process. So just have a weekly update with the engineering manager on any changes not in the spec.”
“Certainly worth looking into Tim. After too much pestering and the engineers having super-tight deadlines, I’ve had it where they would stop responding to my e-mails and calls. So if the engineering managers are more open to giving us updates on changes, it is worth a try.”
-------------
“So with all of these suggestions, do you think we will be able to meet our goal? Specifically, to be able to ship a product that meets the QA customer release criteria by the November ship deadline?” Tim asks, confirming that all of the key areas have been sufficiently addressed.
“I feel better about that possibility, Tim. Much better,” both Nir and Anthony indicate. Tim updates the Excel list accordingly.
Sequence the Intermediate Objectives (IOs)
Tim looks at Nir and
Anthony, “So based on our discussions, I captured and labeled the following suggestions:
- A. Look at ways to complete some of the engineering software modules earlier.
- B. Develop more detailed test plans that have high-level test passes, detailed test passes, how long it takes to complete each pass, and how long it takes to complete a complete test cycle.
- C. Set up a simple smoke test that needs to be completed before checking code into QA and have the engineering managers sign off on the idea of the test.
- D. Use the more detailed test plans to better quantify to management the time needed for testing.
- E. Set up a formal signoff sheet to document testing that management wants to skip tests due to time constraints.
- F. Work to set up a weekly meeting with engineering managers to review code changes not captured in the spec.
It is not too bad of a list, but we cannot do all of them at once. We should go through and see which ones make sense to do first. It would also be good to specify an owner and a timeline for each suggestion as well. Anthony, based on the list, would [A] look at ways to complete some engineering software modules earlier come before [B] develop more detailed test plans?”
Anthony indicates, “They both can pretty much be done in parallel.”
“I can follow up on looking at ways to complete software modules early over the next few weeks since I am meeting with the engineers,” Tim suggests.
“Nir and I can work on the test plans. It will take us about a month, but we are pretty early in the test process, so there is time.”
“Okay, Anthony, so for [C] set up the smoke test, would it come before [A] complete some modules early or [B] develop more detailed test plans, or would they be done in parallel?” asks Tim.
“Again something in parallel. I can look at it over the next week or so. Given that I was burned by it pretty bad on Griffin, I have a pretty clear idea of what we need. Once it is done, Nir and I can meet with the engineering managers.”
“Sounds good. So would [D] use the test plans to better quantify to management the time needed to do testing go after [B] develop more detailed test plans?” asks Tim.
“We need the plans first, and then we can use them to help level set management on how long it takes to test. Once we are pretty comfortable with the test, I can bring it up with the regular bi-weekly meeting I have with some of the management team. After that I can look at bringing it in for a formal manager review,” Anthony explains.
“Okay Anthony, looking at the Excel sheet, this is what I have so far for our plan. A, B, and C can be done in parallel. D can be done only once we have completed B.”
“What about [E] set up a formal sign-up sheet when management wants to skip tests due to time constraints?” inquires Tim.
“That one is a little touchy,” Nir says, jumping in. “Definitely after [B] set up the test plans, probably better after [D] get management to see how long the test plans are and what they involve. I can set up the basic form. Anthony, since you are working with the managers, can you take this one?”
“That is fine,” states Anthony. “I will bring it up as we do the manager review of the test plans. If they know they might be asked to sign off on something, they will be a lot more attentive about what the test plan says.”
“The last one—[F] work to set up a weekly meeting with the engineering manager to review code changes?” Tim continues.
“I can take that one,” Anthony volunteers, “Since I already talk with the engineering manager. I can meet with him over the next month or so. In regards to your question, we should be able to do that in parallel with the other tasks.”
“So, looking at the Excel sheet, this is what I have for our plan,” Tim explains.
“Looks like a good start, Tim. What if we find additional items later on?” inquires Nir.
“Duh,” states Anthony, “We just add them to the plan.”
“Correct Anthony,” confirms Tim, “We can continue to build out the plan
as needed. We do not want to get too detailed, but we do need to cover the key areas.”
Results
“So Nir and Anthony,
before we wrap up this meeting, what do you think about the process we went through and the plan we developed?,” asks Tim.
“Well Tim, to be honest, based on Griffin, I was feeling pretty burned out and seeing the whole test process as pointless,” states Nir. “Sure, the test plans helped in the beginning, but at the end we just had to wing it. I was sure Phoenix was just going to be another repeat of Griffin. Going through this exercise, I feel a bit more optimistic and empowered. There will be issues, but at least I feel like there is something we can do to try ahead of time to improve the situation.”
“Actually Tim, I like the fact that you could take Nir’s endless complaining and put it to good use,” states Anthony. “All kidding aside, you let him vent his issues and once we were done with that, we turned around and started to constructively look at what we could do to address the issues he had raised.”
“The part I liked Tim is that you did not just complain or dictate a plan. Anthony and I were involved and able to share our different concerns and insights. Overall, we provided the detail and helped to put the plan together. We have ownership of it.”
“I’m glad I could help,” states Tim. “Now I need to catch up with Gary and see how he is doing.”
FRT
Just as Tim leaves, he pulls out his FRT diagram and adds one more box: 13) we improve our quality assurance testing.
