If reviews are important to you, then ideally you’re going to want to test your Android app on something very close to the reviewer’s machine to see if it works correctly or not. There are all sorts of things that can go wrong: odd behavior on different versions of the Android OS, not handling hardware or software buttons correctly, not having a fluid enough design to handle large and small device screens, not testing on Wi-Fi, not accounting for network speed variations across different carriers. The list is endless.

OpenSignal produced a report in July 2013 (http://opensignal.com/reports/fragmentation-2013/) that put the number of Android devices in the wild at just under 12,000 running on eight different Android operating systems, and the number is obviously growing. So there are now so many devices and configurations out there that testing across all possible variations isn’t realistic.

In the past it was a reasonable strategy to test on a small sample of phones and a small sample of tablets to get a decent cross-section of all the combinations. But there is now such a variety that separating these devices into phones, tablets, and even phablet categories is just too simple an approach for our testing purposes.

Perhaps if we go back to the OpenSignal report and look at the usage statistics by brand then we might have more success. The graphic in Figure 7-1 tells us that a little over 50% of devices are made by Samsung.

Looking at the OpenSignal data a bit closer we find that a Galaxy S4, SIII, SII, S, Y along with a Note and Note II with a Google Nexus thrown in for good measure covers about 20% of the market. But that’s still only 20% of the available devices.

And what about different Android operating systems? We probably also want to test on multiple Android operating systems. Google’s chart of different Android OSes hitting the Google Play Store in the seven days leading up to November 1, 2013 can be seen in Figure 7-2.

At the very least your app should support Ice Cream Sandwich and Jelly Bean, which have a combined total of 72% of Android users. And there is a strong argument to support Gingerbread, which comes in at 26.3% of Android users. However, supporting Gingerbread or even Froyo comes with some development tradeoffs, such as finding ways to support Action Bars in these earlier versions. Earlier versions of the phones typically also have much less CPU power and memory and can behave radically differently.

Choosing a Strategy

We know we’re not going to test on 12,000 devices, so we need to figure out a strategy to identify issues that our users may experience before releasing the app. We can separate these options into the following choices:

• Test using the devices from our Samsung/Nexus short list.
• Test using devices with each generic screen size.
• Test using devices running each of the major Android OS.
• Find a specific target market, such Kindle or Nook users.
• Test more devices using the Android emulator.
• Test more devices using manufacturers’ emulators.
• Use a third-party testing service.
• Borrow devices from a manufacturer.
• Do crowd-testing.

I’ve talked about the short list of devices in the previous section, and if that suits your budget and needs, then something as simple as that might work very well for you. But it’s not exactly capturing the market 80/20; it’s more like 20/80. And of course it’s always going to be a moving target, so for most people that’s just not going to work. If you’re lucky you might be able to limit it to Ice Cream Sandwich and above, or only tablets, or if you’re really lucky to a specific target market, such as only Kindle tablets.

Alternatively, you can buy devices with small, normal, large and extra-large screen sizes and with low, medium, high and extra-high density. And these could be carefully chosen so that they’re running different Android OSes to get a better cross-section of tests. If that’s your approach, you might want to try using the Android emulator, which provides just such combinations. We’ll see in the next section that you can also download additional emulators to test against more devices. There are also other third-party emulators; some are free (Samsung) and some are paid services (Perfecto Mobile).

Manufacturers are also aware of these issues, and some manufacturers, such as LG, are allowing developers to rent out devices for short periods of time.

Finally, there are some crowd-testing options, such as TestFairy, where your social media friends and family beta test on their Android devices and TestFairy organizes the testing so you can see videos of your users testing.

You need some criteria to help you make your decision, such as these:

• Budget
• Testing automation
• Supported OS
• Supported devices

In this chapter we’ll explore these options in more detail so you can make informed decisions after you form your own criteria.

Emulators

If you’re like most developers your budget can probably stretch to buying or borrowing at most a half dozen devices, which is close to our original scenario. On a limited budget, if you want to test on more devices, there is always the Android emulator, shown in Figure 7-3.

The Android Virtual Device manager( AVD) ships with the following default devices:

• Nexus 4
• Nexus 10
• Nexus 7
• Galaxy Nexus
• Nexus S
• Nexus One
• 10.1 WXVGA (Tablet)
• 7.0 WSVGA (Tablet)
• 5.4 FWVGA
• 5.1 WVGA
• 4.7 WXGA
• 4.65 720p
• 4.0 WVGA
• 3.7 FWVGA
• 3.7 WVGA
• 3.4 WQVGA
• 3.3 WQVGA
• 3.2 QVGA
• 3.2 HVGA slider
• 2.7 QVGA slider
• 2.7 QVGA

As you can see, it’s a mixture of Google Nexus devices and a number of older generic devices. Because the emulator is also notorious for being very a slow and inefficient way of testing, it would seem to be not much use other than for some initial alpha testing.

But there are some simple steps you can take to extend the AVD to make it a lot more useful:

• Install the Intel x86 Atom System Image
• Create your own device
• Install manufacturers add-ons
• Test with multiple emulators in Jenkins

Hardware Testing

Many people don’t want to test on emulators and would much prefer to test on real devices. No matter how good the emulator software is, there are many other factors that might impact how your application may perform in the field that an emulator simply cannot test, such as how you app behaves on different carriers or if you need to make an actual call as part of your app.

There are some options open if you must test on real hardware, besides simply buying all the devices you need:

• Use a third-party testing service.
• Borrow devices from a manufacturer.
• Do crowd-testing.

Summary

This chapter has shown some ways you can address the problem of testing your applications across the hugely fragmented world of Android devices, without needing to bankrupt yourself purchasing 12,000 or more different pieces of hardware. With clever targeting of a subset of devices, using manufacturer AVDs and third-party device testing services, you can greatly magnify your device testing footprint.