Device Testing
If reviews are important to you, then ideally you’re going to want to test your Android app on something very close to the reviewer’s machine to see if it works correctly or not. There are all sorts of things that can go wrong: odd behavior on different versions of the Android OS, not handling hardware or software buttons correctly, not having a fluid enough design to handle large and small device screens, not testing on Wi-Fi, not accounting for network speed variations across different carriers. The list is endless.
OpenSignal produced a report in July 2013 (http://opensignal.com/reports/fragmentation-2013/) that put the number of Android devices in the wild at just under 12,000 running on eight different Android operating systems, and the number is obviously growing. So there are now so many devices and configurations out there that testing across all possible variations isn’t realistic.
In the past it was a reasonable strategy to test on a small sample of phones and a small sample of tablets to get a decent cross-section of all the combinations. But there is now such a variety that separating these devices into phones, tablets, and even phablet categories is just too simple an approach for our testing purposes.
Perhaps if we go back to the OpenSignal report and look at the usage statistics by brand then we might have more success. The graphic in Figure 7-1 tells us that a little over 50% of devices are made by Samsung.
Figure 7-1. Android brand fragmentation
Looking at the OpenSignal data a bit closer we find that a Galaxy S4, SIII, SII, S, Y along with a Note and Note II with a Google Nexus thrown in for good measure covers about 20% of the market. But that’s still only 20% of the available devices.
And what about different Android operating systems? We probably also want to test on multiple Android operating systems. Google’s chart of different Android OSes hitting the Google Play Store in the seven days leading up to November 1, 2013 can be seen in Figure 7-2.
Figure 7-2. Android operating systems using Google Play
At the very least your app should support Ice Cream Sandwich and Jelly Bean, which have a combined total of 72% of Android users. And there is a strong argument to support Gingerbread, which comes in at 26.3% of Android users. However, supporting Gingerbread or even Froyo comes with some development tradeoffs, such as finding ways to support Action Bars in these earlier versions. Earlier versions of the phones typically also have much less CPU power and memory and can behave radically differently.
We know we’re not going to test on 12,000 devices, so we need to figure out a strategy to identify issues that our users may experience before releasing the app. We can separate these options into the following choices:
I’ve talked about the short list of devices in the previous section, and if that suits your budget and needs, then something as simple as that might work very well for you. But it’s not exactly capturing the market 80/20; it’s more like 20/80. And of course it’s always going to be a moving target, so for most people that’s just not going to work. If you’re lucky you might be able to limit it to Ice Cream Sandwich and above, or only tablets, or if you’re really lucky to a specific target market, such as only Kindle tablets.
Alternatively, you can buy devices with small, normal, large and extra-large screen sizes and with low, medium, high and extra-high density. And these could be carefully chosen so that they’re running different Android OSes to get a better cross-section of tests. If that’s your approach, you might want to try using the Android emulator, which provides just such combinations. We’ll see in the next section that you can also download additional emulators to test against more devices. There are also other third-party emulators; some are free (Samsung) and some are paid services (Perfecto Mobile).
Manufacturers are also aware of these issues, and some manufacturers, such as LG, are allowing developers to rent out devices for short periods of time.
Finally, there are some crowd-testing options, such as TestFairy, where your social media friends and family beta test on their Android devices and TestFairy organizes the testing so you can see videos of your users testing.
You need some criteria to help you make your decision, such as these:
In this chapter we’ll explore these options in more detail so you can make informed decisions after you form your own criteria.
Emulators
If you’re like most developers your budget can probably stretch to buying or borrowing at most a half dozen devices, which is close to our original scenario. On a limited budget, if you want to test on more devices, there is always the Android emulator, shown in Figure 7-3.
Figure 7-3. The Android Virtual Device (AVD) emulator
The Android Virtual Device manager( AVD) ships with the following default devices:
As you can see, it’s a mixture of Google Nexus devices and a number of older generic devices. Because the emulator is also notorious for being very a slow and inefficient way of testing, it would seem to be not much use other than for some initial alpha testing.
But there are some simple steps you can take to extend the AVD to make it a lot more useful:
Install Intel x86 Atom System Image
The Android Emulator can take 3-5 minutes or even longer to start and has been the source of a lot of frustration for Android developers. If you’re running on an Intel PC or Mac, then installing the Intel x86 Atom System Image will make the emulator start in 1-2 minutes. Figure 7-4 shows how to install the Intel accelerator from the Android SDK Manager. Note that it must be installed for each Android API level.
Figure 7-4. Installing the Intel x86 Atom system image
Once it is installed, the system image appears as a drop-down choice for the CPU/ABI in the AVD, as you saw in the earlier AVD setup screen in Figure 7-4.
Create Your Own Device
Although the result doesn’t have the device skin, creating your own emulator version of a device is pretty straightforward. For example, take the following steps to create a Galaxy S4 device. (The dimensions for the S4 came from Wikipedia, http://en.wikipedia.org/wiki/Samsung_Galaxy_S4.)
Figure 7-5. Creating an S4 device
Figure 7-6. Creating an S4 AVD
Now while the emulator is only a facsimile of the real thing, if you launch your app it will fit the screen and behave in the same way as an S4.
Downloading Manufacturer’s AVDs
It’s not always necessary to create your own AVDs, as many of the Android device manufacturers provide their own AVDs and even skins for you to download and install. For example, Amazon provides a number of Kindle device definitions. Take the following steps to install Kindle devices:
Figure 7-7. User-defined sites for add-on AVDs
Figure 7-8. Adding Kindle Fire packages in the SDK Manager
Figure 7-9. Adding Kindle Launcher packages in the SDK Manager
You can now either create a Kindle AVD in the Android Virtual Device Manager or use the Amazon AVD launcher, which you can find in the sdk/extras folder; see Figure 7-10.
Figure 7-10. Amazon AVD Launcher
Figure 7-11 shows an example Kindle HDX AVD in action.
Figure 7-11. Kindle HDX AVD
Table 7-1 lists a number of the device manufacturers offering emulator add-ons.
Table 7-1. Android Add-On Device Providers
Manufacturer |
URL |
---|---|
Sony |
http://dl-developer.sonymobile.com/sdk_manager/Sony-Add-on-SDK.xml |
Amazon |
|
Nook |
|
HTC |
Automating Emulator Testing with Jenkins
By now you should be able to create or install AVDs for as many target devices as you think you need. Table 7-2 provides a list of AVDs so you can get good coverage of the current Android marketplace; it comes from another Android AVD repository, this time at https://github.com/j5at/AndroidAVDRepo.
Table 7-2. List of AVDs for Emulator Testing
Smartphones |
Tablets |
---|---|
Nexus One, by Google |
Kindle Fire, by Amazon |
Nexus S, by Google |
Kindle Fire HD (7" and 8.9"), by Amazon |
Galaxy Nexus, by Google |
Galaxy Tablet, by Samsung |
Nexus 4, by Google |
|
G1 aka Dream, by HTC |
|
Droid (original), by Motorola |
|
Droid X, by Motorola |
|
Droid Razr Maxx HD, by Motorola |
|
Galaxy S2, by Samsung |
|
Galaxy S3, by Samsung |
|
Galaxy S4, by Samsung |
Creating these AVDs now allows you to test your Android APK against the most popular devices that are currently available. However, to do this manually would be really time-consuming, and it’s also going to be an ever changing list as new KitKat phones are released. We need a way to automate this testing to be sure that we’re testing any changes we make to our APK.
We the first used Jenkins in Chapter 4, while implementing Agile Android techniques. In this chapter we’re going to use it to automate our builds using as many emulators as we need as Jenkins will allow you to set a variable for multiple emulators.
Take the following steps to automate the builds using multiple emulators.
Figure 7-12. Multi-configuration Jenkins project
Figure 7-13. Adding multiple emulators to the configuration matrix
Figure 7-14. Adding emulator variable
When Jenkins runs on the next build, it will show whether the tests passed on each of the different emulators, as illustrated in Figure 7-15.
Figure 7-15. Multiple emulator test results
Many people don’t want to test on emulators and would much prefer to test on real devices. No matter how good the emulator software is, there are many other factors that might impact how your application may perform in the field that an emulator simply cannot test, such as how you app behaves on different carriers or if you need to make an actual call as part of your app.
There are some options open if you must test on real hardware, besides simply buying all the devices you need:
There are a number of online testing services. Some of these sites are free, and some are paid services.
As you saw earlier in this chapter, the majority of devices in the market are manufactured by Samsung. They don’t offer add-on emulators but do offer their own emulators in the cloud as a free service, see http://developer.samsung.com/remotetestlab/rtlDeviceList.action and Figure 7-16.
Figure 7-16. Samsung remote test lab
Like all of the cloud testing services, it works by first checking out a device and then uploading the APK to a remote server; then the device’s video output is projected onto your remote emulator. Testing for basic developer accounts is, however, limited to testing on 10 devices per day using a basic free account.
The front-runner in the paid services arena at the moment is Perfecto Mobile, but there are many others. Perfecto also has a plug-in for Jenkins that will allow you to automate the APK testing on multiple devices.
Borrow Devices from Manufacturers
Manufacturers know that device fragmentation is an issue, if not the issue, with testing Android devices. Most have their own developer sites that make some attempt to alleviate the issue. And while Sony and Kindle have created emulator add-ons, and Samsung has its own emulators, LG takes a different approach and will loan you a physical device for your own testing. Currently the devices are as follows, and you can sign up at http://developer.lge.com:
One of the more innovative solutions to solving fragmentation is to use crowd testing. It’s common for developers to beg, borrow, and steal devices for testing from their friends and family. However, because nobody wants to give up their phone, this strategy doesn’t work for anything other than very ad-hoc testing. Crowd testing from companies like TestFairy orchestrates this testing in a much more organized fashion. APKs are sent out to a list of emails or to a LinkedIn or Facebook link, and the testers’ interactions with the app are recorded on video so they can be reviewed later.
Summary
This chapter has shown some ways you can address the problem of testing your applications across the hugely fragmented world of Android devices, without needing to bankrupt yourself purchasing 12,000 or more different pieces of hardware. With clever targeting of a subset of devices, using manufacturer AVDs and third-party device testing services, you can greatly magnify your device testing footprint.
3.136.20.252