CHAPTER 6

Pilots (Series of Successes)

Before any new product or service is deployed to the masses, it is prudent to do a trial run, or pilot project. This allows the organization to determine the viability of bringing the new product or service to market, while also gauging the customers’ appetite for the product, before investing in a full-scale deployment.

As stated in Chapter 1, some innovations are widely accepted by customers, while other innovations that are scientifically superior—for example the Dvorak keyboard—are not accepted by the market. The risk of expenditures for a full-scale deployment should be mitigated by running a pilot trial first.

An example of a pilot trial is the trial of their delivery service in Florida by McDonald’s Restaurants in early 2017. Crain’s Chicago Business reported:

The trial in Florida was expanded to several more cities, so it is a series of escalating pilots. They did a similar pilot in 2016 with their all-day breakfast. That pilot was so successful that it was expanded nationwide and led McDonald’s to their highest stock price ever.

This chapter outlines the selection criteria, location, timing, and employee involvement in the pilot, and ends with some yardsticks to judge success.

Selection Criteria

Most, if not all, large companies have a standard set of procedures to roll out new products. The pilot phase should follow this as much as possible.

Some forethought, though, needs to go into criteria to determine where the pilot should take place, and what a successful pilot would look like. Some of the common criteria to be considered for determining where the pilot should take place are as follows:

Client Base Segment

When choosing a pilot for a new product or service, it is important to look at which segment of your client base you wish to address. Is the new product or service aimed at a particular segment? Does the marketing team want to go across a variety of segments to determine which part of the client base is most interested in the new product or service? Does the marketing team want to try different campaigns in different markets to do a multilayer test strategy?

Different industries use wildly varying pilot strategies. Video game companies, for example, like to pilot new video games by sending out beta versions to their most avid buyers. They usually include most of the new features but not all, and usually don’t charge any money. They look for feedback, and gather market intelligence before rolling out the entire product at full price. Major games, like the recently released Tom Clancy’s Ghost Recon Wildlands, have had very successful rollouts using this strategy.

Another example of piloting to likely buyers is when McDonald’s Restaurants piloted the sale of pizza in their restaurants. They chose my hometown, Ottawa, where pizza is a very popular food item. They ran the pilot for several months, but discovered that the length of time it took to cook a pizza from scratch—seven minutes—was very long for most of their clients. And many clients would change their order since their pizza would take too long. Their efforts to predict demand and reduce wait time seemed to be unsuccessful, and they ended the pilot without a major rollout to the rest of the country.

If you are going to choose just one segment of your client base to target for the pilot phase, and you are looking for the best results, target the segment that is most likely to buy the new products or services based on previous buying habits.

What Level of Quality Should Your Product Have in the Pilot Phase?

Many companies in Silicon Valley live by the motto, “Fail fast and fail often.” Most companies should not adopt this motto when running a full-blown pilot. If a pilot is to predict the market’s appetite for a new product or service, the best possible quality must be presented. If there is time pressure to meet deadlines while there remain issues with the product, it is best to label the pilot phase as a beta program and charge accordingly.

Today’s consumers have high expectations in terms of quality of a product or service. With social media allowing most people the ability to rate and review any product or service, it’s important that a new product or service is not seen as low quality. Many customers often search the Internet for feedback on new products and services, and it’s imperative that the feedback is all good for new products to survive.

Physical Location

Online retailers and companies that sell exclusively to online retailers may not need to worry about where to run pilot programs as much as businesses with presence in every major U.S. market. Online retailers generally have systems set up that can ship their products anywhere in the world, so restricting their location is nearly impossible. Bricks-andmortar companies, though, do need to consider which location is best for running a pilot program.

If there is general market acceptance in most locations, then any location could be chosen at random. The success or failure in the location chosen is likely to be the same at any other location. However, when the company’s acceptance is more guaranteed in certain locations, then the area with generally the highest acceptance should be chosen. If the product succeeds here, it is not guaranteed it will succeed everywhere else; but if the product fails here, then it is likely not to work elsewhere. If a midmarket area were chosen, then the product’s success or failure would not be as indicative across other segments.

Pilot Size

Generally speaking, the larger the pilot, the closer the resemblance will be between pilot results and full-deployment results. As a risk mitigation strategy, though, the pilot should be as small as possible to keep investment low until the ability of the new product or service to meet the market needs is proven.

A general rule of thumb for the pilot phase of a new product or service is to cover 10 to 20 percent of the full market. This will be a small enough investment so that total failure would not affect the bottom line, while success at this level should give enough confidence that full-scale deployment should be initiated.

Pilot Duration

The Seinfeld television series started with an original slate of four shows. After the first four shows, ratings were near the bottom of ratings of all television series on air at the time. It was quite conceivable that the show could have been cancelled due to its low ratings, and probably would have been cancelled if it were to start airing in today’s competitive environment. However, the show continued past its initial slate and was able to continually grab more and more market share until it was number one in the market.

Hindsight is 20/20. It is easy to look back and say it was a good decision to continue longer with the innovative show to see if it could eventually garner a large market share, which it did. How long should your pilot phase be in order to determine if your new product or service will be a success or not?

There is some dependency on the product or service you are bringing to market. For example, if you were to introduce a new air-conditioning product in December in a northern climate, it will probably not catch on. Similarly, introducing a new air-conditioning product in May should not be stretched beyond any reasonable purchasing time frame.

As with location, it is best to target a time frame that will give you the greatest chance for success. Back-to-school items, for example, should be piloted in July and August when the market is looking mainly for such products.

If your new product or service has no favorable time frame, then it is necessary to select a time period that you think will represent a good sample of buyer interest. It should be long enough to help you adjust to customer feedback (like the McDonald’s pizza pilot trial in Ottawa, Canada) to see if there is a slight variance that can dramatically improve the odds of success for your new product or service.

Recommended Investment in Pilot

If the Infectious Innovation Process has been followed step by step, where ideas have been selected from a large sample and then vetted through the business case and senior management backing phases, your company should have the confidence to make a significant investment in the pilot phase of this new product or service. Most of the above criteria will have an effect on the investment needed to run the pilot, but the actual money spent needs to be monitored and focused on as well.

Investing in a pilot project needs to be significant enough to guarantee the likely outcome of the product if it were to advance to a full-scale deployment. Marketing budgets need to be scaled to meet the demand of a pilot, as well as the manufacturing costs, where applicable. The intent of the pilot is to determine if the forecast ROI will reasonably be met.

The Number of Internal Resources

The number of internal resources needed to run the pilot may influence the decision on where to run the pilot. If there is a significant number of internal resources needed to run the pilot, then it may be a good idea to select a location which is close to where a significant number of staff members work. Otherwise, the travel and accommodation costs may have an adverse effect on the return.

The seven selection criteria need to be weighed against each other. Nothing works in isolation. On the other hand, successful businesses already have many of these things figured out. They will already have a significant work force located near the biggest customers. Hence, the targeted client base segment, physical location and location of internal resources is probably straightforward if your business is sticking within your most successful product lines. If new product lines are being targeted, then it might not be such an easy choice.

Consider the selection criteria, but don’t over analyze! Make your choice and move forward. There are many rewards for those who take the calculated risks.

Judging Success

When investments are made in pilots, there should be some performance criteria put in place in advance. Sales, meeting customer demand, and product returns are possible performance criteria to put in place. The number of sales measures success in promotion, pricing, and placement, as well as success in the sales process of educating the sales force and having the sales force be able to put that education to good use. The ability to meet customer demand measures success of the manufacturing and operations teams to produce the products and services in a timely manner. Product returns and customer complaints measure the lack of quality in a product. Success is typically measured by the lack of customer complaints and product returns, so there’s an inverse relationship.

Expected values for sales, ability to meet customer demand, and product returns should be recorded as a baseline. As the pilot trial is run, actual values should be compared to the expected values. If the actual values are not meeting expected values, adjustments should be made. Sometimes all the adjustments in the world are not going to be able to help the organization meet its goals. In this case, there needs to be a decision to stop the pilot.

On the other hand, some pilots are wildly successful and far exceed customer expectations. A great example is the Magic Kingdom in Walt Disney World in Orlando. Disney bought 43 square miles of swampland in central Florida, and opened in 1971 with one theme park and three hotels, covering less than 10 percent of the purchased land. As the theme park became more successful, additional hotels and theme parks were added on the land. Currently, there are four theme parks, two water parks, four golf courses and twenty-eight Disney hotels on the property.

Pilot versus Rapid Prototyping

There is a lot of buzz around the term rapid prototyping. This is not the same as a pilot trial. Rapid prototyping is a set of techniques to create a model of the end product in a short period of time. When applied to software, rapid prototyping refers to creating a scale model of the software to see how it would work. This might address the technical issues, and allow senior management to see how the new product or service would look or act, but it is far from a pilot trial.

A pilot trial is much more comprehensive than rapid prototyping. A pilot trial includes sales and marketing, operations, and finance to run through the design, manufacture, sales, and distribution of new products and services. It is creating an entire ecosystem to develop, sell, and support a new product or service, but on a smaller scale than the entire market. For instance, a pilot trial of a new medical device might involve the design, development, manufacture, sales, and distribution to a large city that is representative of the expected market for the device.

Having the ability to develop and manufacture the device does not ensure the market will accept it. Having a market with a hungry appetite for the product or service you are developing does not ensure the market will rush to buy your product or service. Get real products or services developed. Put them in front of real customers, and see how it works.

With a pilot trial, adjustments can be made. Sometimes the marketing needs to be beefed up to attract the right clientele. Other times, the new product or service needs another feature or two. Pilot trials allow organizations to test on a small scale before rolling out to everyone.

Large organizations may consider a city to be a small scale, while others consider a country to be a small scale. Use discretion with the criteria listed below to determine how big your pilot should be, and how far it should stretch.

Having your organization run a pilot trial is much different from having your client ask for a pilot trial to test the product. When a client requests a pilot trial of your product, it normally indicates a lack of trust in the ability of your organization to deliver on the quantity and quality expectations.

Other times, a sales force will offer to do a free pilot trial to get the client hooked on the product. I once worked for a small start-up that was trying to acquire large financial institutions as clients. They offered to do pilot trials in an effort to show some potential clients how valuable their software was. They didn’t charge the client for the software at first, while doing trials to determine what the most popular features were and how to tweak the software to best meet the client’s demands. The financial institutions were happy to try the software for a limited time, but never purchased the product.

When pilot trials are discussed with respect to the Infectious Innovation Process, the intent is to run through the sales cycle as it is intended to be.

Alternatives

In modern times, there are many alternatives to pilot trials. Rather than roll out a full deployment to a test market, organizations are using different methods to determine if the features of a product are technically robust, while also determining if market demand is being met.

Focus Groups

Television shows have long used focus groups. They will show an episode or two to a focus group to gain their feedback on how popular the show may be. This is more important today than ever before. The number of scripted television series has exploded over the last few years. In 2010, there were 216 scripted television series, while the number more than doubled to 455 in 2016. This is due to many organizations taking advantage of the technological innovation of streaming video to meet an increasing market demand to binge on entertainment. With mobile devices, customers may be entertained many more hours of the day than they could have been years ago.

Focus groups can be held on a large scale. When living in Ottawa, I was invited to a focus group with 500 others to watch two different television shows. If 500 people were watching in Ottawa, there were probably thousands across North America invited to attend for their feedback.

This doesn’t mean focus groups are always right. In early 2017, Pepsi produced a commercial that showed Kendall Jenner leaving a modeling gig and walking into the middle of a standoff between protesters and police. She offers one of the police officers a Pepsi, and the protesters and police all celebrate. Focus groups liked the ad, and gave feedback that the majority of the viewers were more likely to buy Pepsi after seeing the ad. When the ad was shown on television, there was tremendous backlash against the portrayal of protesters and the interaction they were having with the police. Pepsi pulled the ad the first day it aired.

Beta Testing

Beta is the second letter in the Greek alphabet and follows the letter alpha. Alpha testing is usually internal testing to test the technical and business aspects of the product, while beta testing is external testing to test the technical aspects on a larger scale and to gauge market demand for the business aspects.

Many software companies, including those in the video game industry, will ask a number of their loyal customers to beta test their software when the software is essentially complete. This allows the software company to test on a larger scale and ensure that the technical glitches are fixed before mass commercial production. It also allows the software company to determine if the features in the product are complete and meet customer demand.

Free Versions

Many developers of software applications for cell phones and tablets sell their product by giving away free versions. The free version will have limited functionality or be funded by advertising. The paid version will have all the functionality or no advertising. Market interest can be gauged by the number of users of the free version who choose to upgrade to the paid version.

Pilot Trials of New Processes

Much of the above discussion has surrounded new products and new services. This involves a lot of exterior contact—contact with suppliers, contact with distributors, contact with retailers, and contact with clients. When pilot trials are run on new business processes, the majority of work is done internally within the organization.

This requires a different mindset for the pilot phase.

The size of a pilot for a new process often seems much easier to control, but is open to office politics, especially when several different departments are involved.

In order to determine the effect of the new process, there often needs to be a baseline of the current process. The AS-IS process is often mapped, and measurements of time, resources, and materials are often taken. To map these processes, there needs to be some consideration into the maturity of the process. Processes are most mature when they are well understood across the organization and different people are following the same processes. Processes are the least mature when everyone figures out for themselves how to get things done.

If the process is deemed to be mature, then a business process engineer only needs to speak with one or two people to map out the AS-IS process. Measurements may be taken in one location and extrapolated to the rest of the organization.

However, when the processes are not mature, then the business process engineer needs to speak with several people to determine how the majority of people follow the process. Measurements need to be taken in many places and averaged out over the organization.

Once the AS-IS processes are mapped and measured, the new TO-BE processes have to be mapped out. These process maps should include who is doing what, and what information is being stored at each stage.

When the AS-IS and TO-BE processes are mapped out, there is usually a gap analysis done to determine where the gaps are between what is being done now and what needs to be done to optimize the situation.

With the gap analysis in place, the business process engineer usually develops and documents an action plan to bridge the gap. This action plan will detail a number of steps to get from the AS-IS to the TO-BE. With a mature process, this is straightforward, and the pilot may be performed anywhere. With an immature process, there needs to be some consideration of where the pilot trial of the new process will take place. Will it be done where the process now takes the longest amount of time, in order to determine the best increase in value for the new process? Or will the pilot of the new process be done where the process now takes the least amount of time, in order to determine the most conservative amount of savings?

If the current process is mature, then running the pilot should go smoothly, since everyone is used to performing the process the same way. However, if the AS-IS process is not very mature, then the pilot will have to take into account the fact that there is no consistency in place. In addition to the steps needed to move to the TO-BE state, there will also need to be steps in the action plan that serve to enforce the new process, and to provide change management services to ensure the staff involved in the process are ready for it.

When the action plan for the TO-BE process is put into effect for the pilot phase of a new process, measurements need to be taken. What resources, labor, and materials are needed to run the new processes? How does the cost of the new process compare to the cost of the AS-IS process? Often, the process will need to be performed several times to get to a steady state. The measurements should not be taken until the process is in a steady-state phase, since initial measurements will not accurately reflect the costs of the TO-BE process as there is always a learning curve to get to the new process.

The success of the pilot phase of the new process can be measured by comparing the costs of the new process once it reaches steady-state mode to the costs of the AS-IS process (which is normally already in its steady-state mode).

Summary

Selection Criteria

Location

Timing

Who’s Involved

Judging Success