Types of Testing

Functional testing. Functional testing is a standard way of testing an app. It’s called functional because we’re testing the app’s features (also known as functionalities) as they are specified in the requirement specification—the requirement specification is something you or a business analyst would have written during the planning stages of the app. The requirement specifications would have been written in a document (usually called functional requirement specification). Examples of what you might find in a functional specification are “user must log in to the server before entering the app,” “user must provide a valid email for registration,” and so on. The testers, usually called QA or QC (short for quality assurance and quality control, respectively), are the ones who carry out these tests. They will create test assets, craft a test strategy, execute them, and eventually report on the executions’ results. Failing tests are usually assigned back to the developer (you) to fix and resubmit. What I’m describing here is a typical practice for a development team that has a separate or dedicated testing team; if you’re a one-person team, the QA will most likely be you as well. Testing is an entirely different skill, and I strongly encourage you to enlist other people, preferably those who have experience in testing, to help you out.

Performance testing. You could probably guess what this type of testing does just from its name. It pushes the app to its limits and sees how it performs under stress. What you want to see here is how the app responds when subjected to above-normal conditions. Soak testing or endurance testing is a kind of performance testing; usually, you leave the app running for a long time and in various modes of operation, for example, leave the app for a long time while it’s paused or at the title screen. You’re trying to find here how the app responds to these conditions and how it utilizes system resources like the memory, CPU, network bandwidth, and so on; you will use tools like the Android Profiler to carry out these measurements.

Another form of performance testing is volume testing; if your app uses a database, you might want to find out how it will respond when data is loaded to the database. What you’re checking is how the system responds under various loads of data.

Spike testing or scalability testing is also another kind of performance testing. If the app depends on a central server, this test will usually raise the number of users (device endpoints) connected to the central server. You’d want to observe how a spike in the number of users affects the user experience—is the app still responsive, was there an effect on frames per second, are there lags, and so on.

Compatibility testing is where you check how the app behaves on different devices and configurations of hardware/software. This is the situation where AVDs (Android Virtual Devices) will come in handy; because AVDs are simply software emulators, you don’t have to buy different devices. Use the AVDs whenever you can. Some apps will be difficult to test reliably on emulators; when you’re in that situation, you have to fork over money for testing devices.

Compliance or conformance testing. If you’re building a game app, this is where you check the game against Google Play guidelines on apps or games; make sure you read Google Play’s Developer Policy Center at https://bit.ly/developerpolicycenter. Ensure you are also acquainted with PEGI (Pan European Game Information) and ESRB (Entertainment Software Rating Board). If the game app has objectionable content that’s not aligned with a specific rating, they need to be identified and reported. Violations could be a cause for rejection, which may result in costly rework and resubmission. If you’re collecting data from the users, you might want to audit the app to check if it is compliant with applicable data privacy regulations.

Localization testing is essential, especially if the app is for global markets. App titles, contents, and texts need to be translated and tested in the supported languages.

Recovery testing. This is taking edge case testing to another level. Here, the app is forced to fail, and you’re observing how the application behaves as it fails and how it comes back after it fails. It should give you insight into whether you’ve written enough try-catch-finally blocks or not. Apps should fail gracefully, not abruptly. Whenever possible, runtime errors should be guarded by try-catch blocks; when the exception happens, try to write a log and save the app’s state.

Penetration or security testing. This kind of testing tries to discover the weaknesses of the app. It simulates the activities that a would-be attacker will do to circumvent all the security features of the app; for example, if the app uses a database to store data, especially user data, a pen tester (a professional who practices penetration testing) might use the app while Wireshark is running—Wireshark is a tool that inspects packets; it’s a network protocol analyzer. If you stored passwords in clear text, it would show up in these tests.

Sound testing. If your app uses sounds, check if any errors are loading the files; also, listen to the sound files if there’s a cracking sound and so on.

Developer testing. This is the kind of testing you (the programmer) do as you add layers and layers of code to the app. This involves writing test codes (in Java as well) to test your actual program. This is known as unit testing. Android developers usually perform JVM testing and instrumented testing; we’ll discuss unit testing more in the following sections.

Unit Testing

Unit testing is a functional testing that a developer performs, not the QA or QC. A unit test is simple; it’s a particular thing that a method might do or produce. An application typically has many unit tests because each test is a very narrowly defined set of behavior. So, you’ll need lots of tests to cover the complete functionality. Android developers usually use JUnit to write unit tests.

JUnit is a regression testing framework written by Kent Beck and Erich Gamma. You might remember them as the one who created extreme programming and the other from Gang of Four (GoF, Design Patterns), respectively, among other things.

Java developers have long used JUnit for unit testing. Android Studio comes with JUnit and is very well integrated into it. We don’t have to do much by way of setup. We only need to write our tests.

JVM Test vs. Instrumented Test

If you look at any Android application, you’ll see that it has two parts: a Java-based and an Android-based one.

The Java part is where we code business logic, calculations, and data transformations. The Android part is where we interact with the Android platform. This is where we get input from users or show results to them. It makes perfect sense to test the Java-based behavior separate from the Android part because it’s much quicker to execute. Fortunately, this is already the way it’s done in Android Studio. When you create a project, Android Studio creates two separate folders, one for the JVM tests and another for the instrumented tests. Figure 13-1 shows the two test folders in the Android view, and Figure 13-2 shows the same two folders in the Project view.

As you can see from either Figure 13-1 or 13-2, Android Studio went the extra mile to generate sample test files for both the JVM and instrumented tests. The example files are there to serve as just quick references; it shows us what unit tests might look like.

A Simple Demo

To dive into this, create a project with an empty Activity. Add a Java class to the project and name the class Factorial.java ; edit this class to match Listing 13-1.

public class Factorial {

  public static double factorial(int arg) {

    if (arg == 0) {

      return 1.0;

    }

    else {

      return arg + factorial(arg - 1);

    }

  }

}

Listing 13-1

Factorial.java

Ensure that Factorial.java is open in the main editor, as shown in Figure 13-3; then, from the main menu bar, go to Navigate ➤ Test. Similarly, you can also create a test using the keyboard shortcut (Shift + Command + T for macOS and Ctrl + Shift + T for Linux and Windows).

Right after you click Test, a pop-up dialog (Figure 13-4) will prompt you to click another link—click Create New Test, as shown in Figure 13-4.

Right after creating a new test, you’ll see another pop-up dialog.

In the window that follows (shown in Figure 13-5), create the new test.

❶	You can choose which testing library you want to use. You can choose JUnit 3, 4, or 5. You can even choose Groovy JUnit, Spock, or TestNG. I used JUnit4 because it comes installed with Android Studio.
❷	The convention for naming a test class is “name of the class to test” + “Test.” Android Studio populates this field using that convention.
❸	Leave this blank; we don’t need to inherit from anything.
❹	We don’t need setUp() and tearDown() routines, for now, leaving them unchecked.
❺	Let’s check the factorial() method because we want to generate a test for this.

When you click the OK button, Android Studio will ask where you want to save the test file. This is a JVM test, so we want to keep it in the “test” folder (not in androidTest). See Figure 13-6. Click “OK.”

Android Studio will now create the test file for us. If you open FactorialTest.java , you’ll see the generated skeleton code— shown in Figure 13-7.

❶	The file Factorial.java was created under the test folder.
❷	A factorial() method was created, and it’s annotated as @Test. This is how JUnit will know that this method is a unit test. You can prepend your method names with “test,” for example, testFactorial(), but that is not necessary; the @Test annotation is enough.
❸	This is where we put our assertions.

See how simple that was? Creating a test case in Android Studio doesn’t involve that much in terms of setup and configuration. All we need to do now is write our test.

Implementing the Test

JUnit supplies several static methods that we can use in our test to make assertions about our code’s behavior. We use assertions to show an expected result, which is our control data. It’s usually calculated independently and is known to be true or correct—that’s why you use it as a control data. When the expected data is returned from the assertion, the test passes; otherwise, the test fails. Table 13-1 shows the common assert methods you might need for your code.

Table 13-1

Common assert methods

Method	Description
assertEquals()	Returns true if two objects or primitives have the same value
assertNotEquals()	The reverse of assertEquals()
assertSame()	Returns true if two references point to the same object
assertNotSame()	Reverse of assertSame()
assertTrue()	Tests a Boolean expression to see if it’s true
assertFalse()	Reverse of assertTrue()
assertNull()	Tests for a null object
assertNotNull()	Reverse of assertNull()

Now that we know a couple of assert methods, we’re ready to write some tests. Listing 13-2 shows the code for FactorialTest.java.

import org.junit.Test;

import static org.junit.Assert.*;

public class FactorialTest {

  @Test

  public void factorial() {

    assertEquals(1.0, Factorial.factorial(1),0.0);

    assertEquals(120.0, Factorial.factorial(5), 0.0);

  }

}

Listing 13-2

FactorialTest.java

Our FactorialTest class has only one method because it’s for illustration purposes only. Real-world codes would have many more methods than this, to be sure.

Notice that each test (method) is annotated by @Test. This is how JUnit knows that factorial() is a test case. Notice also that assertEquals() is a method of the Assert class, but we’re not writing the fully qualified name here because we’ve got a static import on Assert—it certainly makes life easier.

The assertEquals() method takes three parameters; they’re illustrated in Figure 13-8.

❶	The Expected value is your control data; this is usually hard-coded in the test.
❷	The Actual value is what your method returns. If the expected value is the same as the actual value, the assertEquals() passes—your code behaves as expected.
❸	Delta is intended to reflect how close the actual and expected values can be and still be considered equal. Some developers call this parameter the “fuzz” factor. When the difference between the expected and actual values is higher than the “fuzz factor,” then assertEquals() will fail. I used 0.0 here because I don’t want to tolerate any kind of deviation. You can use other values like 0.001, 0.002, and so on; it depends on your use case and how much fuzz your app is willing to tolerate.

Now, our code is complete. You can insert a couple more asserts in the code so you can get into the groove of things if you prefer.

There are a couple of things I did not include in this sample code. I did not override the setUp() and tearDown() methods because I didn’t need it. You will typically use the setUp() method to set up database connections, network connections, and so on. Use the tearDown() method to close whatever it is you opened in the setUp().

Now, we’re ready to run the test.

Running a Unit Test

You can run just one test or all the tests in the class. The little green arrows in the gutter of the main editor are clickable. When you click the small arrow beside the class’ name, that will run all the tests in the class. When you click the one beside the test method’s name, that will run only that test case. See Figure 13-9.

Similarly, you can also run the main menu bar test and go to Run ➤ Run.

Figure 13-10 shows the result of the test execution.

Android Studio gives you plenty of cues so you can tell if your tests are passing or failing. Our first run tells us that there’s something wrong with Factorial.java; the assertEquals() has failed.

Tip

When a test fails, it’s best to use the debugger to investigate the code. FactorialTest.java is no different than any other class in our project; it’s just another Java file; we can debug it. Put some breakpoints on your test code’s strategic places, and then instead of “running” it, run the “debugger” so you can walk through it.

Our test failed because the factorial of 1 isn’t 2, it’s 1. If you look closer at Factorial.java, you’ll notice that the factorial value isn’t calculated correctly.

Edit the Factorial.java file, then change this line

return arg + factorial(arg - 1);

to this line

return arg * factorial(arg - 1);

If we rerun the test, we see successful results, as shown in Figure 13-11.

Instead of yellow exclamation marks, we now see green checkmarks. Instead of seeing “Test failed,” we now see “Test passed.” Now we know that our code works as expected.

Instrumented Testing

Unit testing that interacts with the Android platform is known as instrumented testing; we will use the Espresso framework to do this.

Espresso is a testing framework for Android to make it easy to write reliable and concise user interface tests. Google released the Espresso framework in 2013. Since its 2.0 release, Espresso became part of the Android Support Repository.

Espresso automatically synchronizes your test actions with the user interface of your application. The framework also ensures that your activity is started before the tests run. It also lets the test wait until all observed background activities have finished.

Recording Espresso Tests

Android Studio includes a feature where you can run your app, record the interaction, and create an Espresso test using the recording. Go to the main menu bar, then Run ➤ Record Espresso Test, as shown in Figure 13-12.

After choosing the “Record Espresso Test,” you can now interact with the app, like usual, but this time, the interaction is recorded. If you click one of the buttons, say the “HELLO” button, the test recorder screen will pop up, as shown in Figure 13-13.

❶	This section shows you each interaction you had with the app. At this point, I clicked the app once only; I clicked the “HELLO” button.
❷	This section is the ViewMatcher but done visually. If you click the TextView, like what I did here, it goes over as an item to the “Edit Assertion” section.
❸	The TextView is selected here because I clicked it in the ViewMatcher section (item 2).
❹	This is where you choose the assertion. In this case, we’re merely using the “text is.”
❺	This shows the actual value of the TextView we’d like to assert.

You can click “Save and Add Another” if you’d like to add another test or “Save Assertion” and finish the recording. Figure 13-14 shows the next screen.

When you click OK, the recorder will prompt the class’ name, where it will save the recording as a test class, as shown in Figure 13-15.

When you go to the src/androidTest folder, you’ll find the newly generated test class from your recording. You can now run the generated test, the same way you ran MainActivityTest earlier.

Note

Two factoids about Espresso: (1) The Espresso recorder is one of the most used tools when using Espresso, according to Android Studio analytics; (2) Espresso recorder was originally named “cassette.”

Summary