In this final, brief chapter, we address what’s coming towards us in the mobile app testing world, at ever increasing speed. The mobile world will grow rapidly, so we must build flexibly, and be ready to change. We must plan for the future and we must try to see around corners as the future comes towards us.

CHAPTER SECTIONS

Chapter 5, Future-Proofing, contains the following four sections

CHAPTER TERMS

There are no terms to remember for Chapter 5.

1 EXPECT RAPID GROWTH

For the foreseeable future, expect rapid expansion of apps, devices, and user expectations. As mentioned in the previous chapter, there are already around six million mobile apps across all the different platforms. As long as the vendors find ways to avoid the dreaded commoditization trap by adding new features and as long as users are willing to pay extra money for snazzy new features like heart rate monitors and who knows what next, you will see continued expansion. As long as money grows on the mobile trees, new vendors will enter the market and partnerships of various kinds will be formed with traditional vendors and traditional brands.

For example, if you are so inclined—and sufficiently wealthy—you can buy an Apple Watch that was made in partnership with Hermés, the famous French company. So, you have a luxury brand that teams up with a technology company, with some interesting testing implications. As another example, consider the Vertu luxury mobile phones, some of which cost over $10,000 and feature things like titanium cases and so forth. Imagine if a large percentage of your customer base consists of people who spend over $1,000 on a smart watch or over $10,000 on a fancy mobile phone, both of which will be technologically obsolete in two years.

The app makers are getting ever cleverer about getting and keeping people’s attention, so people are constantly using their devices. This means that expectations about reliability, usability, and performance will only go up.

We will probably also see the reality of—and people’s expectations of—a convergence in the look and feel of mobile apps. For example, consider that, in the early days of ecommerce, Amazon pretty much set the standard for how ecommerce is supposed to work, and if you tried to do something that was very different, unless it offered noticeable advantages to the customer, that would be resisted.

Not only are device vendors trying to avoid commoditization, but app makers also must fight harder and harder to try to distinguish themselves as well, so features are added quickly and fixing bugs immediately is the order of the day.

So, user expectations are high already and those expectations will only get higher. What we have to do, as testers, is to try to understand and cover those expectations as much as we possibly can within constraints.

5.1 Test your knowledge

Let’s try one or more sample exam questions related to the material we’ve just covered. The answers are found in Appendix C.

Which of the following statements is true?

In the future, users of mobile applications will:

2 BUILD FOR CHANGE

Faster, better, cheape—back in the late 1990s, the head of NASA, Dan Goldin, pushed this concept. I’ve always scoffed at that with the response, “Sure. Pick two.” There were some NASA missions that went very well, but there were also some missions that flopped, such as a couple of failed Mars missions.

Like it or not, in mobile apps for the near future, due to the competition, the need to stay relevant, and the need to avoid becoming a commodity, faster, better, cheaper is a reality we’ll have to live in. So, you need to find ways to use tools, automation, and good automation architectures. You’ll need to make smart decisions about buying versus renting tools and environments. As one reviewer of this book mentioned, this also means being as involved as possible in project planning so you can plan your testing accordingly.

You’ll need to keep your skills sharp, so that you are as efficient as possible. After all, if you’re in a faster, better, cheaper kind of world, efficiency is really the name of the game, more so than effectiveness. A good job done in a day will beat a perfect job done in a week, and an okay job done in an hour beats the good job done in a day.

What does this mean to us as testers? Well, if we can get to 75 percent coverage or just 50 percent coverage in two days, but it will take a month to get to 100 percent, we have to go for the 75 percent or even the 50 percent. That’s a tough pill to swallow. Many times, as testers, we want to champion a philosophy that proclaims there ain’t no substitute for doing it right. As much as I would love to be able to hold to that, I think for certain mobile apps you’ll find that there is a substitute for doing it right, which is doing it okay and quickly.

It depends on what you’re working on. If you’re working on a safety-critical application where, if that application gives bad information or makes a bad thing happen, somebody will die, then faster, better, cheaper is not a good motto. However, faster, better, cheaper does apply to you if you’re making a disposable app that will be around for six months or so. These things come and go like pollen, so do you really want to sweat the last detail of quality? Probably not.

However, there are certainly mobile devices where you would sweat the last detail of quality. For example, do you want a faster, better, cheaper pacemaker implanted in your chest? I have clients that work in the FDA-regulated world, and they’re not the faster, better, cheaper types. If I said, “Faster, better, cheaper,” to them, they would look at me like I had two heads.

How to build for change

Part of building for change is having a flexible test architecture.¹ Think Lego®. A flexible test architecture is Lego®-like, and consists of test components that are Lego®-like. This means things like many small scripts can be snapped together to do many different things.

Part of this will be enabled by having the right tools, so again I’ll stress the need to be very careful with tool selection. As I’ve mentioned, I’ve seen many times that sloppy, hurried tool selection was the first chapter in a sad story about how, in some cases, over a million dollars was wasted on automation that didn’t work out.

Flexible test architectures are also enabled by test maintainability. Don’t take shortcuts on maintainability, either. Use people who have the skills and the smarts. Also, try to use people who have done it before, in the sense that they’ve built automation solutions and environments in the technological and business spaces that you’re working in.

Be smart and have a “just diverse enough” environment. What I mean here is that you use real devices where you have to, but avoid drowning in the cubic meter of devices.

Use risk-based testing to make sure that you are focused on the most important coverage areas, as described earlier in this book. As one of the reviewers of this book mentioned, make sure you are using risk-based testing appropriately, based on the type of app you’re testing and the risks associated with it.

Look for ways to take advantage of crowdsourcing, testing in the wild, and testing in production, but carefully manage any business or quality risks associated with those options. (As an example of what I mean by quality risks, keep in mind that, in some cases, such as a safety-critical app that controls an insulin pump, such testing may border on criminally reckless.) For example, when testing in production, you can use what’s called A/B testing. In A/B testing, there are features that are out there and they’re available on your server, but only a small percentage of your users actually see them. You set up your app to select certain users (but not all) based on certain characteristics of their use and expose them to new features. You evaluate their response to the new features, and decide based on that whether to roll the features out to more users, and see how that works.

As I said, efficiency is king in the faster, better, cheaper world. So, know your testing return on investment and your automation return on investment, and look for ways to optimize those. It will be less about effectiveness and more about efficiency when you’re time-squeezed, unless you’re in the safety-critical world.

Further, when you’re thinking about investment in various test assets such as tools, automated tests, environments, data, and so forth, consider the app. If you are in the business of building disposable apps that are intended to be a flash in the pan, something people use for about six months or so before moving on to something else, spending a lot of money creating a really sophisticated test automation system for such an app is foolish.

So, for such apps, look for the quick and cheap not the slow and perfect. Now, that said, whilst the test automation part might not be a good place to invest, your environments and tools might be things you want to invest in, assuming that those will transfer from the current disposable app to the next one.

A recurring theme in this book is that you must be really careful in your selection of your test tools, your test partners, and your testing service providers. There are good discussions of tool selection criteria in the ISTQB® Certified Tester Foundation syllabus and the Advanced Test Manager syllabus 2012 which can provide guidance in this regard.²

In addition to the factors discussed in those documents, you need to keep in mind that things are moving more rapidly in the mobile world. This means that vendor and/or open-source community responsiveness and tool flexibility should figure high on your priority list when you put your requirements together. And I do mean this literally, in that you should have a prioritized requirements list as part of your tool selection process.

Another consideration, as I mentioned, is the expected app lifetime. If you’re building a throwaway app, use a throwaway test automation approach. Perhaps a quick set of automated tests running through a simulator gets you some decent benefit, quickly.

So, you can compromise some of your requirements on the tools and compromise on your automation goals, if it’s a matter of urgency and kind of a throwaway situation. However, in terms of vendors, partners and so forth, you want long-term relationships with trusted partners that value you as a business partner.

Here’s something to keep in mind about testing service providers and tool vendors of all sorts. When you’re evaluating these companies, sometimes organizations take the viewpoint that big is better. They want to use some huge company with a huge name that is a market leader in terms of revenue.

However, before you do this, consider the amount of business that you will do with them. In order for you to be an important customer to one of the top ten testing service or tools companies, you must do millions and millions of dollars of business with them every year. If that’s not your budget, you’ll never be a really important client to them.

This means that the company won’t assign its top talent, its star players, to your projects or account. If it later happens that one of the people they assign to your projects grows into a star player, that person will probably be moved to another project or account. So, if you intend to spend a limited amount of money, the small to mid-size providers might be a better choice, if you’re looking to get a testing service provider.

Now, that same rule doesn’t always apply exactly to tool selection. A smaller tool vendor might be willing to work harder to earn your money and they might be willing to be more responsive to your needs. However, you have to balance that against the possibility that the smaller tool vendor will go out of business.

For both service providers and tool vendors, you’ll want to examine their track record, including how responsive they have been to change and how they’ve handled disruptive technologies. For example, you might ask them, “Suppose the wearables mobile market catches fire tomorrow and we start focusing on apps for those platforms, explain how you could move quickly enough to help us test such apps.”

Let’s look at an example of the old saying that the perfect is the enemy of the good. Suppose I have two potential approaches to automation. One will allow me to address 75 percent of my coverage goals; it’s a GUI and keyword-driven approach, and it’s something that will be maintainable for years. The other only gets to 25 percent of my coverage goals, and works through APIs and the data layer.

Now, if these are the same cost, require the same effort, and have the same breakeven point, I will take the first one. However, what if I can go through the API and data layer and have that done in a third of the time or less? This is not a purely hypothetical question, because API automation is relatively easy, tends to be relatively maintainable, and allows access to a lot of the business functionality through APIs in many cases. So, that might be good enough.

It’s also possible that you don’t do either; you do both. This exact situation came up with a client a while back. What I told them was to attack part of the test automation problem through the API and get what they could that way. Then, attack another part of the test automation problem through reports comparisons, which is basically going to the data layer, and testing through reports is relatively easy too. Once they had those two quick and cheap partial coverage approaches in place, they should look at what they hadn’t covered and think about a keyword-driven approach that allowed them to achieve the rest of their coverage goals over a longer period.

5.2 Test your knowledge

Let’s try one or more sample exam questions related to the material we’ve just covered. The answers are found in Appendix C.

To allow for change in the future, you should:

You are planning the testing of an application that must work on devices with many parallel applications running. The application must gather images and weather readings from devices accessible via the local area network and the internet, and process that data in real time. For the first release, your company intends to support Android devices only, as your main customer for the application has already standardized on Android smartphones and tablets.

Which of the following should be part of your strategy for reducing test environment maintenance costs?

3 PLAN FOR THE FUTURE

Software engineering has experienced a number of big trends, to the point where the Gartner analysts have created something called the hype cycle. If you’ve never seen the hype cycle, a quick search on the internet will locate it.³ It’s an easy-to-understand concept, which says that trends start small, but then reach a point where they get hyped all out of proportion to what they actually offer. Ultimately, the hype subsides and the trend—whatever it was—becomes absorbed into the general way software engineering is done.

Here’s an example of a software engineering hype cycle. In the late 1980s and early 1990s, object-oriented programming became a major force. Some people said and wrote that object-oriented programming would create a world where we would have huge collections of pre-tested objects, written in C++ or some other similar object-oriented language. These pre-tested objects would be like common civil engineering materials, such as I-beams, concrete, steel plates, asphalt, and so forth. Software engineering would become a simple matter of snapping these together, assembling them like building blocks or Lego®, to build whatever you want.

Since these were all going to be pre-tested objects, all of our quality problems would disappear. Since these would be handy building blocks available to everyone, productivity would go through the roof.

Well, none of that happened. Consider that Microsoft Windows 10 is written in large part in C++. I would not hold that up as a paragon of quality. What we saw was that the reality of object-oriented software did not live up to the hype.

In Software Testing Techniques (1990),⁴ Boris Beizer made a joke about this kind of thinking. The title of the section in the book is, in Latin, the language will be our savior, thus poking fun at the quasi-religious levels of fervor that have surrounded such fads.

If this were an isolated incident, it would be an amusing anecdote but there’d be little to learn from it. However, it’s not isolated, but rather is just a particularly large example of something that happens all the time. Fourth generation languages were another example of a language fad.

Total Quality Management and Six Sigma are two more examples of ideas that became fads, though I know that there are very valuable concepts in both of them. In the 1990s, the Software Engineering Institute took ideas from Total Quality Management, especially the idea that the quality of the product arises from the quality of the process. So, they created an approach to building software, expressed in the Capability Maturity Model or CMM, that is focused on implementing best practices of software engineering throughout the software process.

Some people might see Agile as a reaction against the approach expressed in CMM and its successor CMMI. However, it’s really an affirmation of the core belief that getting the process right is central to software success. To rephrase Beizer’s aphorism, software engineering is in the grips of a long-term fixation on process as the savior.

However, life cycle processes continue to change, with Kanban and DevOps the emerging ideas now. These process changes include good ideas and bad ideas. Often, one of the main drivers behind these changes is accelerating the delivery of software. Back when Agile was emerging as a major force, I remember closely questioning some executives at a conference about what Agile really was. I got a lot of amorphous answers, so I kept pushing. It finally came down, always, to a simple goal: we want the software faster.

So, your challenge will be explaining to managers and executives how there is such a thing as enough testing and such a thing as not enough testing. You also have to explain that tools are not magic, but involve hard work, skills, and expertise.

You will have an ever-increasing number of mobile testing resources available, in terms of testing service providers, tools, cloud testing, crowdsourcing, and so forth. There will be new and expanded ways to involve the end user in the testing process.

However the process changes, these tools, these resources, while potentially helpful, will not replace smart testers making smart testing decisions.

As long as we continue to believe, as an industry, that getting the process right will save us, that the process is the key to speedy delivery, that the process is the key to quality, we will continue to see life cycle churn. The same pattern will repeat. Last year’s newest and greatest process ideas will be revealed as not actually being magic bullets, only to be replaced by the next great idea in the software engineering process.

Eventually someone will realize that improving the software engineering process is just part of the puzzle. Clearly, having an organized, carefully designed life cycle will result in better software. However, even a perfect life cycle won’t solve all of the challenges of software engineering.

5.3 Test your knowledge

Let’s try one or more sample exam questions related to the material we’ve just covered. The answers are found in Appendix C.

In the future, mobile software development life cycles will:

4 ANTICIPATING THE FUTURE

So, as I’ve said, you should expect processes to continue to churn. New tools will bring change. Testing practices will continue to change, especially in the dynamic world of mobile development, as well as for other emerging technologies such as robotics, the Internet of Things, artificial intelligence, virtual reality, self-driving cars, and 3D printing. However, remember that the best practices that have been established in testing for years and years abide, although they are often ignored.

For example, a very large number of people get paid to be professional testers. Less than half of those testers, however, could sit down and properly apply equivalence partitioning to design a test for an input screen. Equivalence partitioning is the most basic fundamental of black-box test design techniques.⁵

This matters because, when people actually do know how to apply fundamental black-box test design techniques, it makes a big difference. I have had clients tell me that, by applying equivalence partitioning to their testing, they increased their efficiency by over 50 percent.

So, learn to apply these proven best practices. Some people believe that mobile technologies change everything, but this is the hype cycle again. Mobile technologies don’t change everything. Proven testing best practices still work, but the way you apply those proven best practices does change.

Over the previous chapters of this book, I’ve talked quite a bit about investments in tests, test data, tools, test automation, test environments, and the like. These are important decisions, and should be made in a thoughtful way. So, try to plan out two years, three years, or further, depending on the size of the investment that you’re going to make in your test environments.

Another theme in this book has been the fierce competition in the mobile devices and apps markets. On the device side, we see a relentless attempt by the mobile device makers to avoid commoditization. I don’t have access to device-maker executive strategies, but I have to believe that many of them understand what happens when you get commoditized because they have been in or continue to be in commoditized markets, or have just paid attention to what happened with PCs. Whatever the specific motivations are, the pace of innovation and new feature deployment on the device-side will likely remain rapid.

The competition is a little different on the app side. On the device side you have a few dozen device manufacturers, and it can only grow so much. There are natural limits to the growth in device manufacturers. There are some pretty significant barriers to entry, at least so long as the components resist commoditization and cannot be bought as easily as, say, the components that make up a PC or laptop.

In the case of apps, there are around six million apps on the market already. Not every app is made by a different company, so there aren’t four million app-making companies, but there are a tremendously large number of companies creating mobile apps. The barriers to entry are entirely non-existent. Anybody can download a mobile OS software development toolkit (SDK), start building apps, and put them in an app store. So, the competition there is all about differentiation in an already-commoditized mobile app market. The competition is likely to continue to be as brutal as a Dickensian orphanage. That kind of competition is no fun from the inside, but it tends to take Schumpeter’s economic concept of creative destruction to new highs, and it will drive a lot of change.⁶

This means that you must be ready to learn a lot of new stuff in a short period of time. You’ll need to take some chances and take some smart risks. If you make mistakes—and you probably will—be sure to learn from them. When you take risks, take calculated ones where you can find out whether the risk is going to pay off quickly.

For example, if you take risks with test automation, do it in such a way that you’ll discover quickly whether your automation approach will actually work and have a positive return on investment. Remember the cautionary tales I told you about companies finding they had a negative return on investment after years of working on their test automation. You don’t want to go through that experience.

If you’re going to try something that fails, you want to be able to recognize it as a failure quickly. This is what’s sometimes referred to as failing fast. Unfortunately, sometimes this fail-fast mantra gets used to justify the software version of throwing half-cooked spaghetti at the wall—an excuse for trying wild ideas with little chance of success. Remember, it’s not true that it doesn’t matter how many times you fail. Frequent failure is not efficient. Ready, fire, aim is how you shoot a machine gun, which is fine if you don’t care about the fact that you’re paying a dollar a round. You get more efficient results with a scoped rifle, where a single well-aimed shot hits the bullseye and where failure to hit the bullseye results in adjustments to the scope, producing a better-placed shot next time.

So, if you miss the bullseye, adjust your scope and try again. How do you adjust your scope? Part of it is assessment of what went wrong. Part of it is also learning from other people’s successes and failures. Do industry research. Study industry trends. Read books on testing, test automation, mobile technologies, and so forth. Learn to write scripts and apps. Learn to program. Learn how to use new testing tools. Read technical publications that discuss emerging technologies, new phones and mobile devices, and so forth.

When things are really moving fast, it really helps to be a geek. By geek, I mean that this is your thing. A geek is someone who gets really excited about whatever it is that geeks them out.

For example, I have a few clients in the video gaming space. I’ve met a lot of game testers. Each of them is a hardcore gamer. It is their thing. They play games at work as part of testing them. They go home from work and play games. They read about games in trade journals and online.

Anytime you’re dealing with something that’s really dynamic and fast moving, it helps if you find the whole subject fascinating and you love immersing yourself in it. If it’s boring to you, and it’s just a job, it will be harder to keep up.

A final word of advice, to expand on something I said earlier. Resist the mindset or slogan of “this changes everything.” It’s just part of the hype cycle, and I see this with some of my clients.

In the early days of Agile, I would hear people say, “Agile changes everything and no traditional testing best practices apply in Agile.” What’s happened over the last 20 years is that people have figured out how to apply traditional testing best practices to Agile, and now people realize that about 90 percent of traditional testing best practices can be and should be incorporated into Agile.

It’s the same thing with mobile. I hear people say, “Mobile changes everything about testing.” Well, it changes some things, for sure, and I’ve discussed some of them in this book.

However, things that come along and change everything, literally everything, these things are really rare. Learning how to extract mechanical energy by burning coal at the start of the Industrial Revolution, yes, that literally changed everything. Learning how to fix nitrogen by making ammonia, thus freeing ourselves from a dependency on bird guano as a source of nitrogen, yes, that literally changed everything. Learning how to split the atom, yes, that was a game changer for sure.

However, if we are talking about computer-related advances that might change everything, something that qualifies for that breathless phrase is the concept of convergence, the loading of human consciousness into a computer. Yes, that really will change everything, because that’s going to be the end of human mortality and that is pretty discontinuous.

Mobile changing everything? Yeah, sure, right, just like Agile changed everything. Whenever I hear people saying “X changes everything,” my immediate thought is that we have somebody pushing some hype. And, being the cynical guy I am, I ask myself, “How are they making money off the hype?”

So, soldier on, and apply and adapt existing testing best practices to mobile testing whilst learning how to apply the new best practices that are emerging for testing mobile apps. To paraphrase the aphorism, have the serenity to accept the things that remain the same, the courage to master the things that do change, and the wisdom to know the difference.

5.4 Test your knowledge

Let’s try one or more sample exam questions related to the material we’ve just covered. The answers are found in Appendix C.

As a mobile tester, you will need to:

In this final, brief chapter we addressed the rapid growth you must expect in the mobile app testing world. We saw that flexibility and readiness for change are key. In addition, successful mobile app testers plan for the future, and try to see around corners as the future comes towards them.

1 Some good information on flexible test architectures can be found at: www.istqb.org/component/jdownloads/send/48-advanced-level-test-automation-engineer-documents/201-advanced-test-automation-engineer-syllabus-ga-2016.html

2 For more details, see my 2012 book Foundations of Software Testing (Cengage: Boston, MA), with Dorothy Graham and Erik van Veenendaal, and my 2009 book Advanced Software Testing: Volume 2 (second edition 2014, San Rafael, CA: Rocky Nook).

3 As of the time of writing, you can find the hype cycle here: www.gartner.com/smarterwithgartner/top-trends-in-the-gartner-hype-cycle-for-emerging-technologies-2017/

4 New York: Van Nostrand Reinhold.

5 There are no industry studies to back up this statistic, but I have spent the last 20 years delivering training as well as other consulting and expert services to clients around the world and I’ve seen well over half of testers unable to apply the equivalent partitioning technique prior to my training them how to do it.

6 You can find a good explanation of creative destruction, or “Schumpeter’s gale,” here: www.econlib.org/library/Enc/CreativeDestruction.html