4

VALIDATION

Validation is all about closing the feedback loop to collect and measure data. By analyzing the data, you get information that turns into knowledge that can be leveraged for future benefit.

Since value is about delighting the customers who consume and buy your products, it raises this question: When do you actually deliver value?

Does a grand plan have value? Maybe indirectly to internal people who are creating the product. But does it have any value to customers? A plan is just an assumption that the prescribed path will lead to value.

Two famous military leaders and one boxer sum this idea up nicely:

Eisenhower’s distinction between planning and plans is even more important in today’s globalized economy where everything moves faster and faster. To be successful, you need to validate your business assumption as quickly as possible. Eric Ries calls this the value hypothesis.¹ All ideas are nothing but hypotheses until customers validate them by paying for and using the resulting products or services. This is called validated learning. When you think along these lines, every new product feature becomes an experiment.

“Eighty percent of the time you/we are wrong about what a customer wants” is the realization from Microsoft.² The company that can fail and learn the fastest on an ongoing basis will be the most successful and sustainable. As an entrepreneur you need to develop the capability to deliver value quickly and learn from your mistakes (or false hypotheses) as quickly as possible.

Former Cisco CEO John Chambers described this in his last public speech as CEO: “Forty percent of businesses in this room, unfortunately, will not exist in a meaningful way in 10 years. 70% of companies will attempt to go digital but only 30% of those will succeed. If I’m not making you sweat, I should be.”³

It is critical to identify and communicate with all stakeholders from the beginning to ensure that you build the best possible product.

Scrum provides an excellent opportunity for stakeholder feedback with the Sprint Review, which creates transparency around the Increment. Stakeholders are the key audience for this event, where you incorporate their feedback into the Product Backlog. Putting your product in front of your stakeholders at least once a Sprint provides these opportunities:

1. Engaging stakeholders. Implementing stakeholder feedback each Sprint helps stakeholders feel listened to and will have them coming back for more.

2. Correcting course. The more often stakeholders see the product, the less you risk going in the wrong direction for too long.

3. Creating accountability. The more transparency you create, the harder it is for stakeholders to hide behind ignorance.

A good example of stakeholder validation is a Subway sandwich shop. Subway creates transparency between the sandwich creation and the customer (see Figure 4-1):

1. Engaging stakeholders. At Subway, you build the sandwich together. The sandwich maker does not go for too long without direction from the customer.

2. Correcting course. Change your mind on sandwich ingredients along the way. “Oooh, those peppers look good, can I have more of those?”

3. Creating accountability. If a customer ends up with a sandwich that doesn’t have the mushrooms they want, whose fault is it? The customer should have said something along the way.

Figure 4-1 Transparent sneeze guard at a sandwich bar

As Product Owner, your focus must always be on getting the product to its next shippable state—a Minimum Viable Product (MVP).

If your hypotheses prove to be wrong, you have not lost too much money and can still investigate other options—or pivot as Eric Ries calls it. This also helps to avoid the “trap of sunk costs,”⁴ the tendency to irrationally follow through on an activity that is not meeting expectations just because of the time and/or money already spent on it. Poker players call this “pot committed.”

Hypothesis validation should be a part of your Product Backlog and can be considered active risk management. The higher the impact, the higher the priority order should be and the earlier you get to inspect at a Sprint Review and Retrospective.

To properly validate the hypothesis, of course, you need to put the right measurements in place. As discussed in the preceding chapter, there are many metrics from which to choose. Avoid using the more circumstantial vanity metrics.⁵ Keep your metrics value neutral—objective/free from opinion—so that they tell you the real story. Face your challenges and work through them one after the other.

Figure 4-2 Kano model

These features are grouped into three categories:

Basic: Essential features that customers assume are included with the product. A product without these basic features would come as an unpleasant surprise to customers and likely not sell, resulting in negative value. A car’s basic features, for example, include the engine, seat belts, steering wheel—everything you’d expect to see in a car.

Performance: Features that customers ask for as they are not always assumed to be included. They carry your customers over the satisfaction threshold from “good enough” toward “very satisfied.” The more performance features you can offer, the better. A car’s performance features are air conditioning, stereo system, navigation system, and some of the more gimmicky things that make driving fun.

Excitement: Features that your customers have likely not even thought about, but are thrilled once they discover them. Excitement features get people talking about your product. A car’s excitement features include seamless smartphone integration, massage seats, self-parking, or even a bird’s-eye view when parking (see Figure 4-3).

Figure 4-3 Jaguar’s bird’s-eye car view

Once you cover the basic features, enough performance features, and a few exciters, you have an MVP (see Figure 4-4).

Figure 4-4 Kano model overlaid with MVP

Because no product is ever “done,” you create a series of MVPs, one after the other. Once the MVP is on the market, you collect your key value metrics, create the next Product Backlog for the next MVP, and so on.

Over time, exciters turn into performance features and eventually into basic features (see Figure 4-5). Anti-lock braking systems (ABS) were once features found only in high-end cars. Now they are considered basic in all cars.

Figure 4-5 Excitement turning into performance and eventually into basic features

For example, before investing months of time and lots of dollars into Bitcoin integration for an e-commerce application, in the very first Sprint you could add a Bitcoin payment option for your users that instead leads to a simple survey. Then you could count how many people actually click on the option and gauge their interest further by asking targeted questions like “Do you have a Bitcoin account?” and “Given the chance, would you pay for these purchases with Bitcoin?” Data like this provides valuable information to determine the validity of your product direction without investing in a complete development effort.

You can also leverage the landing page to gather data about traffic or create a call to action.

Landing pages are typically easy to do build in early Sprints and get the Development Team in the habit of releasing something right off the bat.

The classic example of this is Zappos. The founder posted photographs of shoes found in local stores, putting them on a blog page where people could order them. As orders came in through e-mail, he ran down to the shoe stores to buy them and shipped them off himself. Only after he saw that there was demand did he invest in building out a backend system, inventory, fulfillment, and the other elements of his business.

Figure 4-6 Steps of the scientific method

The step “Publish Results” is where fellow scientists are asked for feedback. That part could take a long time, making the whole feedback loop long.

In Lean Startup, Eric Ries introduces a simplified product-centric version of the scientific method steps with the intent of speeding up the feedback cycle. Ries describes this as the Build-Measure-Learn principle (see Figure 4-7).

Figure 4-7 Build-Measure-Learn feedback loop

As described in Chapter 6, the three pillars of the empirical process control theory fit nicely into this picture: Measure is transparency, learn is inspection, and build is adaptation. This also aligns nicely with the three Vs: ideas as vision, product as (potential) value, and data as validation.

With every product, initiative, or feature there should always be the question, “Have we created enough value?” Empirical process control allows you to inspect and adapt along the way.

At the core of Lean Startup thinking are two activities:

1. Persevere

2. Pivot

When the data validates assumptions or is inconclusive, then you continue to persevere, gathering more data along the way.

When the data reveals that you are not gaining the value originally hoped for, then it is time to cut bait, pivot, and avoid the trap of complacency. Even if the expected value is there, the data could unveil a new, more prosperous path that you must have the courage to pivot toward.

Figure 4-8 summarizes these concepts.

A pivot is about adjusting your path toward your vision or even establishing a whole new vision. Having data available—direct evidence—is helpful, but intuition also plays an important role in coming up with new ideas. Shorter feedback cycles give you courage to follow intuition more often as your ideas will eventually be substantiated through constant validation.

Pivoting is not an easy thing to do. It requires that you accept you have been wrong and that you must adapt. Too often the acceptance of failure is masked by vanity metrics or false loyalty to your product.

Once you accept that you are here to learn as quickly as you can and reduce the time spent between learnings, you will have more opportunities to pivot and hence improve your chances of success.

In 1959 Henry Kremer offered, in today’s value, a prize of $2 million USD for the first person to cross the English Channel by a human powered aircraft. This was achieved 20 years later in 1979 by Paul MacCready. Why so long? It certainly wasn’t a lack of interest. The many previous attempts by competitors spent months and years designing and building their crafts just to see them shatter after seconds into their flight. This fear of failure would weigh heavily on most and result in even longer, “better” plans.

Paul MacCready realized that it was not about the plane itself but about the possibility to build a plane that could be tested and retested within hours. He achieved about four attempts per day, and with his 223rd try, he was successful. The speed of learning was essential to his success.

Ensure that the Vision, Value, Validation feedback loop is closed as often as possible. The higher the frequency, the more often you can pivot and the better your chances of success.

Figure 4-8 Three Vs leading to persevere or pivot decision

Your limited resources provide you with only so much runway (time and budget). By the time you reach the end of your runway, you must have enough speed to take flight. The length of the runway is set, so your ability to pivot is the determining factor for success, or you find yourself crashed at the end of that runway.

____________________________

1. Eric Ries, The Lean Startup (New York: Random House, 2011), 61.

2. Ronny Kohavi et al., “Online Experimentation at Microsoft” (ThinkWeek paper, Microsoft Corp., Seattle, WA, 2009), http://ai.stanford.edu/~ronnyk/ExPThinkWeek2009Public.pdf.

3. Julie Bort, “Retiring Cisco CEO Delivers Dire Prediction: 40% of Companies Will Be Dead in 10 Years,” Business Insider, June 8, 2015, http://uk.businessinsider.com/chambers-40-of-companiesare-dying-2015-6.

4. “Definition of ‘Sunk Cost Trap,’” Investopedia, accessed February 22, 2018, http://www.investopedia.com/terms/s/sunk-cost-trap.asp.

5. Ibid.

6. Noriaki Kano, Nobuhiku Seraku, Fumio Takahashi, and Shinichi Tsuji, “Attractive Quality and Must-Be Quality,” Hinshitsu 14, no. 2 (1984): 147–56.

7. Ries, Lean Startup.

8. Anthony Morris, “A Willingness to Fail Solved the Problem of Human-Powered Flight,” Financial Review, October 16, 2015.