Chapter 15. Using App Center to build, test, and monitor apps
This chapter covers
- Getting started with Visual Studio App Center
- Setting up app builds in App Center
- Running UI tests on real devices in the cloud using App Center
- Capturing and analyzing user data including analytics, behavior, and crash reports
Back in chapter 1 we discussed the mobile-optimized development lifecycle that’s ingrained in all the mobile tools offered by Xamarin and Microsoft. It’s shown again in figure 15.1.
Figure 15.1. The mobile-optimized development lifecycle is a continuously iterating cycle of develop, test, build, distribute, and monitor.
In previous chapters we’ve looked at the first couple of steps in this lifecycle: develop and test. We’ve written apps and built unit tests and UI tests. Now we need to consider the remainder of this lifecycle, and to do this we need to take advantage of another tool from Microsoft and Xamarin called Visual Studio App Center. If you’re excited about DevOps (and I hope you are) then this chapter is for you. If not, this chapter should be relaxing after the intense learning of the previous chapters—it’s more about learning how to configure and use App Center than about learning new concepts. The end product should be worthwhile—a built and tested app set up for crash reporting and user analytics, ready to be released to users in the next chapter.
15.1. Introducing Visual Studio App Center
Visual Studio App Center is billed as “Mission Control for apps,” and it provides a one-stop shop for managing the mobile-optimized development lifecycle in your mobile apps, covering part of test as well as build, distribute, and monitor:
- Test— App Center integrates with Test Cloud, which allows you to take the UI tests you wrote in the last chapter and run them on real devices in the cloud. (In this case, the “cloud” is a warehouse in Denmark with racks and racks of phones and tablets.)
- Build— App Center has tools that can take your code and build it either on demand or when you push to source code control.
- Distribute— From App Center you can distribute your builds to beta testers, including adding notifications inside your app when an update is available. In an upcoming release you’ll also be able distribute to the Google Play and Apple App Stores.
- Monitor— Using App Center you can not only get analytics about the users who are using your apps, you can also get automatically uploaded crash reports if your app crashes for your users.
In this chapter and the next we’ll be looking at using App Center to perform all these steps in the lifecycle, starting in this chapter with build, test, and monitor, and looking at distribution in the next chapter. This is slightly out of order in terms of the lifecycle, but it makes sense to developers—we need to set up the build before we can test it, and we need to add analytics before we distribute so that we can see the results as soon as possible.
You can access App Center at https://appcenter.ms (figure 15.2). At the time of writing, it’s free for light usage, with paid plans if you need more features. You can log in by clicking the Get Started button, and then create an account using either a Microsoft account (you probably set one of these up when installing Xamarin, if you didn’t already have one), or by connecting it to your GitHub, Facebook, or Google account.
Figure 15.2. Visual Studio App Center—Mission Control for apps
Once you’ve connected to App Center, it’s worth setting up an App Center password associated with the email address used by the service you connected with. This will allow you to log in from the command line later, which is necessary for running UI tests. To do this, click the cog icon next to your name to see the user settings, go to the Password tab, and click the Send Set Password Email button. You’ll receive an email with a link to follow to set a password. Click this link and set a password.
15.1.1. Apps
App Center is built on the concept of apps, with each app being a single, releasable app for a single platform. Although you may think of Countr as a single app available on iOS and Android, from the perspective of App Center it’s actually two apps, one Android and one iOS.
App Center covers more than just Microsoft products
As part of the new “Any developer, any platform” approach, Microsoft has made sure that App Center works for all mobile developers, not just Xamarin developers. App Center supports native iOS and Android apps built using Objective-C, Swift and Java, Xamarin, Windows UWP, and React Native—and it actually supported native iOS and Android before Xamarin. This means that if you work with a mixture of app technologies, you can still use App Center to manage everything.
For each app, App Center provides a number of services:
- Build— App Center can connect to a source code repository (currently GitHub, VSTS, and BitBucket are supported). For each branch in your repo you can create a build to compile your solution or an individual project using any available build configuration either on demand or whenever the code is updated in the repo. You can sign your build using an Android keystore or iOS provisioning profile and even launch it on a real device, using Test Cloud as a sanity check to ensure that your app runs and doesn’t crash on startup.
- Test— Xamarin has a huge data center in Denmark with thousands of real devices that you can run your app on using automated UI testing. App Center can be used to configure, start, and display the results of these test runs.
- Distribute— App Center can distribute your app, as soon as it’s built, to beta testers or to the Google Play or Apple App Stores. Beta test users will get an email with a link to download your app so that they can test it, and they can even get notifications inside their apps that an update is available. I’ll cover distribution in chapter 16.
- Crashes— Most apps will crash at some point, and App Center has an SDK you can add to your apps to track crashes. If your app crashes, the App Center SDK will track this, and the next time your app is run and is online, it will upload the crash details, including the stack trace, to App Center.
- Analytics— It’s always good to know how your users are using your app, so that you can track pain points and see what features are popular or are not being used, so you know which areas to work on. It also helps to see user demographics—if your app is in English but is popular in Italy, you might want to offer it in Italian, for example. The App Center SDK can help to track this kind of information.
15.1.2. Users and organizations
When you log in to App Center, you’re logged in as a user, and you can have apps assigned to you as an individual. Users can also be part of an organization—essentially a named group of users with different permissions. You can be an admin user and have full control over the organization, or a collaborator who can just create and manage apps inside the organization. You can read more on organizations in Microsoft’s documentation at http://mng.bz/ge1o.
15.1.3. API
App Center was written API-first—that is, the development team at Microsoft created public APIs for managing your apps, and then the web portal was written to use these APIs. This means that anything you can do in the App Center web portal, you can also do using a public REST API. You can easily integrate App Center into any existing build or CI tools that you already use. For example, if you already have an automated build and release process using a tool like Jenkins, you can replace the steps for building your app with calls to App Center, and use that to run your builds. Microsoft’s API documentation is available at http://mng.bz/szn6.
15.1.4. CLI
App Center also has a command-line interface that you can use to do everything App Center can do from a command line. Microsoft’s App Center docs have more information on the CLI, covering the huge range of features it offers: http://mng.bz/Bkin. We’ll get this set up and use it later in this chapter to run automated tests.
15.1.5. Getting help
App Center has excellent documentation. Just click your name at the bottom of the menu and select Docs & APIs (or you can find it at http://mng.bz/fS43). You can also interact directly with the App Center team at Microsoft if you get stuck, find bugs, or have feature suggestions—just click the blue and white speech-bubble button. From there you can start conversations with the team, see their responses, and see messages about new features.
15.2. Setting up builds
App Center apps are based on code for a mobile app from a source control repository, so before you can set up a build, you need to put the source code for your app into a repository that App Center can access. At the time of writing, App Center supports three providers: Git repos in GitHub (https://github.com) and BitBucket (https://bitbucket.org), and Git or TFS repos in Visual Studio Team Services (https://www.visualstudio.com/team-services/). Source code control is outside the scope of this book, so if Git is new to you, I recommend Learn Git in a Month of Lunches by Rick Umali (Manning, 2015). For the rest of this section, you’ll need to have a basic understanding of Git, including being aware of branches, commits, and pushes.
In this chapter and the next, we’ll just be looking at setting up Countr, but feel free to set up SquareRt as well, for practice. You’ll need to add your Countr code to one of the three source code repository providers. Each one has comprehensive documentation to get you started. I personally use GitHub, because you can use a GitHub account to log into App Center, and this automatically gives App Center access to your repos, but use whichever you feel most comfortable with.
Ideally you need a private repo
The different source code repository providers have both private and public repositories available. Later in this chapter we’ll be adding an app secret to our app to wire up analytics and crash detection, and ideally this value should be kept private. This means it’s worth creating a private repo to put your code into.
15.2.1. Creating your first App Center app
Once your code is in source control, the next step is to create an app in App Center. We’ll begin by creating an Android app, then create the iOS one.
From the App Center landing page, click the Add New App button at the top right.
A panel will slide out on the right where you can enter details for your app (figure 15.3). Set the name as Countr - Android, add a description, set the OS to Android and the Platform to Xamarin, and then click Add New App. This will create the app and show a Getting Started page with details on how to set up the App Center SDK for crash reporting and analytics (we’ll look at that later in this chapter).
Figure 15.3. Setting up your app in App Center
The next step is to configure a build, and to do this you need to connect App Center to your source code repository. Click the Build tab on the menu on the left, select your repository provider of choice (figure 15.4), and follow the instructions to connect it to App Center. Once it’s connected, choose the repo that you put the source code for Countr into.
Figure 15.4. To create a build, you first need to connect App Center to your source code repository.
15.2.2. Configuring the Android build
Once you’ve set up your repo, the Build tab will show all available branches in that repo, so if you set up a new Git repo you’ll probably only see one branch called “master”. Click the master branch, and then click Set Up Branch. A panel will slide out with options to configure your branch, as shown in figure 15.5.
This configuration is divided into four sections:
- Build App— In this section you can configure how your app should be built.
- Project— The first option is the project to build—your solution could contain multiple Android apps, so here you can select the one to build. Our solution only has one Android app, so Counter.Droid should be selected.
- Configuration— From here you can set the build configuration you want to compile with: either Debug or Release (or any others if you have more set up). You’ll need a release build for running UI tests later (to avoid having to uncheck the Shared Mono Runtime option in your Android project), so set this to Release.
- Mono Version— Here you can select which version of Mono (the open source version of the .NET Framework that runs on macOS) to use when compiling
your app. At the time of writing, Mono 5 has been released with a whole swath of improvements and new features, but there
have also been some bugs. You can choose to use the latest version (currently Mono 5) or use Mono 4.
Figure 15.5. App Center can build your branch, as well as sign it, test it, and set it up for distribution.
- Build Frequency— The build frequency can be set to either on every push to this branch, or manually, on demand. Leave this set to Build This Branch on Every Push—this way every time you push one or more commits to your repository, the app will be rebuilt.
- Sign Builds— App Center can sign your builds using your Android keystore. You can take advantage of this to sign your build using the keystore you created back in chapter 13. Turn on Sign Builds, upload your keystore from the very safe place where you keep it, enter the alias you used when creating it in the KEY_ALIAS, and then enter your keystore password into both the KEYSTORE_PASSWORD and KEY_PASSWORD fields (when you set up a keystore using Xamarin, it uses the same password for the keystore as for the signing key inside it).
- Test on a Real Device— By turning this option on, your app will be launched on a real device inside App Center’s Test Cloud service, and the test will verify that your app launched successfully. It won’t run the UI tests you wrote in the previous chapter; it will just launch the app and check for a crash (we’ll look at running UI tests in Test Cloud later in this chapter). This is a great sanity check that your build has worked and that your app at least starts up. You have no control over which device it will run on—it will be a compatible one with at least your app’s minimum SDK installed, but other than that you have no control. It will also make your build take longer, as you’ll have to wait for an available device. This runs using UITest, so use a release build so the shared Mono runtime isn’t used, as discussed in chapter 13. You’ll need a valid Test Cloud subscription for this to work, but as part of your App Center free trial, you’ll get a valid subscription; turn this option on.
- Distribute Builds— These options allow you to distribute final builds to either beta testers or the store. We’ll look at this in chapter 16, so leave this off for now.
Once your build is configured, click Save. This will queue up a first build under the master branch, as shown in figure 15.6.
Figure 15.6. All builds for a branch can be seen by clicking on the branch.
Under each branch is a list of all the builds for that branch, and this will only have one entry so far—the build that was kicked off by setting up the branch. If you click the build, you can see more information about it, including a full build log. Figure 15.7 shows the bottom of the build log.
Figure 15.7. Your Android app should build successfully.
Once your app has built, you’ll see the build marked as a success, as well as how long the build took and logs detailing everything that happened. The build has taken your code from your source code repository, restored the NuGet packages you use, compiled your Android app, signed the APK using your keystore, installed it on a device, and launched it to verify that everything has worked. You’ll also have an option to download the Android APK and build logs if you want to. If your app didn’t build, check the build output and fix up whatever is causing the issue.
This is your Android app set up and built. Now let’s set up the iOS app.
15.2.3. Configuring the iOS build
Although you’re building a cross-platform Xamarin app, you’re really building two apps, one iOS and one Android. This means you have to set up your iOS app as a separate app in App Center. Repeat the same steps as for the Android app, but call the app Countr - iOS and set the OS to iOS.
As before, select the master branch to set up the build. The Build App section has a few different options than the Android version, as shown in figure 15.8:
- Build app
- Project— Unlike Android, you can only select the solution you want to build here, not the project. This means you should only have one iOS app in your solution, to avoid confusion—if you have more than one, it might be worth breaking them out into separate solutions. This setting should default to Countr.sln.
- Configuration— This is the same as the Android configuration section. To be able to run a launch test on your builds using UITest, you need to use the Debug configuration, so set this to Debug instead of Release.
- Mono Version— Here you can select which version of Mono to use when compiling your app. At the time of writing, Mono 5 has been released
with a whole swath of improvements and new features, but also some bugs. Here you can choose to use the latest version (currently
Mono 5) or to use Mono 4.
Figure 15.8. iOS builds have a few more options than Android builds.
- Xcode Version— You can configure which version of Xcode to use when building. Only recent versions are available (for example, at the time of writing 8.3.2 is the latest, and you can only select versions starting at 8.0), but this is useful if you need an earlier version. You can just leave the default setting, which will be the latest released version.
- Build Type— Unlike Android, iOS builds come in two types—simulator and device builds. Set this to Device Build so that you can run a launch test on a real device (launch tests aren’t available on simulators).
- Build Frequency— Leave this set to Build This Branch on Every Push.
- Sign Builds— App Center can sign your iOS builds using your certificate and provisioning profile.
- Provisioning Profile— The easiest way to set the provisioning profile is to head to the Apple Developer site (https://developer.apple.com), log in, go to the provisioning profiles section of Certificates, Identifiers and Profiles, and download your developer profile. Once it’s downloaded, upload it to App Center.
- Certificate— App Center needs your signing certificate as a .p12 file. You can’t download this from the Apple Developer website; instead
you need to export it from your keychain. Open Keychain Access on your Mac and search for a certificate whose name starts
with iPhone Developer (figure 15.9). Select this certificate and click File > Export Items, enter a name, select a sensible location, and click Save. When you’re
prompted, enter a password to use to encrypt this certificate.
Figure 15.9. Use Keychain Access to export your certificate.
- Test on a Real Device— Turn this option on to sanity check your app on a real iOS device.
- Distribute Builds— Leave this off for now.
Once your build is set up, click Save and watch the logs to see it build and run the launch test. Again, if the build fails, check the build output and fix up any issues.
15.3. Testing your apps using Test Cloud
When your apps were built, App Center uploaded them to a real device for a simple launch test. When you write your own UI tests and run them locally, you can run them against simulators or actual devices, but there’s a limit to how many devices you can feasibly own and use. When running UI tests from the IDE, you can only target one device at a time, and although there are ways to run tests on multiple devices at the same time, this requires complicated configuration for Android and a lot of Mac build servers for iOS. Ideally, you’ll want to test on a wide range of devices of all configurations running all manner of OS versions, but this means a massive hardware spend and a long time to run all the tests. Fortunately, Test Cloud offers an alternative.
15.3.1. What is Test Cloud?
Test Cloud is a cloud-based device lab containing many thousands of real Android and iOS devices that you can run your UI tests on, and it’s available as part of App Center. You can create a test session with a selection of devices, submit your tests to it, and it will wait for those devices to become available, install your app on them, and run your UI tests.
You can then see the results in a dashboard, where you can break them down by OS, device type, device manufacturer, OS version, or whatever you need to analyze any test failures. You can also see screenshots of your app running, track memory usage, and even download the device log.
This is a paid service (with a 30-day free trial, of course, to lure you in), but it removes the cost of buying and maintaining your own devices, plus the time of setting up and running tests manually.
15.3.2. Preparing your apps to be tested
Test Cloud runs UI tests against a signed and packaged application: an APK (Android package) for Android or an IPA (iOS application archive) for iOS. Android apps should be release builds, but iOS apps should be debug builds so that the Calabash server is enabled.
There are two ways to get this package: downloading the build from App Center, or building locally. At the time of writing you can’t automatically test a build inside App Center (apart from the launch test). You have to run tests from your local dev box.
Downloading packages from App Center
When App Center builds and signs your app, it creates an app package—an IPA on iOS and an APK on Android. If you select an individual build for a branch, you’ll see a Download button at the top right (figure 15.10). Click this and select Download Build to download the APK or IPA.
Figure 15.10. You can download the packaged build from App Center.
Building locally
To generate the Android APK locally, set the build configuration in Visual Studio to release and run your app on a device or emulator—this will compile your app and package it as an APK.
For iOS, IPAs aren’t built by default for debug builds in Visual Studio, so you need to configure your builds to generate them by checking the Build iTunes Package Archive (IPA) option in the iOS IPA Options section of the project properties (figure 15.11). With this option ticked, you’ll get an IPA every time you build your app, so tick it, set the configuration to debug (remember that you need debug to have the Calabash server enabled), and rebuild. Your IPA needs to be a device build, so make sure it’s built against a physical device (simulator builds won’t work on Test Cloud), and if you don’t have a device at hand, select Generic Device. Don’t worry about which provisioning profile is used for the device build; Test Cloud will re-sign it with their provisioning profile.
Figure 15.11. To create an IPA file with every build, turn on Build iTunes Package Archive.
Compiling the UI tests
You’ll need to make sure your UI test project has been built from Visual Studio. The configuration doesn’t matter. You just need to remember which configuration you used, because you’ll need the path to it later when you run the tests.
15.3.3. Creating a test run configuration
When you want to use Test Cloud, you start by creating a test run configuration, and this configures which OS your app supports and what devices you want your test to be run on. You can then assign tests to a series, allowing you to group multiple test run configurations. For each run configuration, you’re given the command that you can run locally to upload your app to Test Cloud and start the test run.
You can access Test Cloud from the Test tab in App Center for one of your apps. Select this tab, and then click the New Test Run button at the top of the screen. Once you click this, you’ll be presented with a panel containing a wizard that will allow you to configure the test run.
The first panel allows you to select the devices you want to run your UI tests on. This includes a range of devices and OS versions (for example, on iOS there are 160 different combinations ranging from iPhones to iPads and OS versions from 8 up), and you can search this list to find the device and OS configurations you want to run on. You can type in the search box to filter by name, or click the funnel icon to filter by device type, OS, CPU, or other values. If you click the information icon to the right, you’ll see a popup giving detailed device information. For now, just select a couple of devices—any device will do.
Once you’ve selected your devices, you can save your selection using the Save Set button, and you’ll be prompted to enter a name for the device set. This device set will be remembered, and the next time you create a test, you’ll be able to reuse this set or create a new configuration.
After selecting the devices you want, click the Select <x> Devices button (figure 15.12).
Figure 15.12. Selecting the devices to test on
In the next panel (figure 15.13) you can configure the test run details.
Figure 15.13. Configuring the test series and testing framework
- Test Series— Test series are named groups for grouping tests based on whatever criteria you want. You should see two test series: launch-test was created when you selected to run a launch test for your app at build time, and master is a default test series created for you. Select master for now.
- System Language— Your tests can be run on devices configured with one of a small selection of system languages, so if you need your devices to run in a different language, select it here.
- Test Framework— Test Cloud supports a number of frameworks, including the native Apple and Google frameworks, Appium, and UITest. Your tests were written using UITest, so select this option.
When you click Next, you’ll be presented with a final panel with details about your test run and instructions on how to run it (figure 15.14). At the moment, you can’t launch a test run from App Center. Instead, you have to run it from the command line. Don’t click the Done button yet, as the contents of this panel are necessary to run the tests.
Figure 15.14. The last panel shows commands to run to execute the test series.
15.3.4. Running tests from the command line
To run your tests, you’ll need to have Node.js installed—the App Center command-line tools are implemented as a Node.js JavaScript package that can be run from the command line. If you haven’t heard of Node.js before, it’s a JavaScript runtime that can run JavaScript libraries such as the App Center tools, among others. Head to https://nodejs.org and install Node.js before continuing.
Once Node.js is installed, install the App Center tools using the Node.js package manager.
For macOS, launch a Terminal window and run this command:
sudo npm install -g appcenter-cli
On Windows, launch a command prompt as an administrator and run this command:
npm install -g appcenter-cli
Once Node.js is set up, navigate to the root of the Countr solution using the Windows command prompt or macOS terminal.
If you’ve never used the App Center command-line interface (CLI) before, you’ll need to log in. Enter the following command:
appcenter login
This will launch a browser window where you’ll need to log in to App Center. Once you’ve logged in, you’ll be given an access code that you’ll need to copy and paste into the command prompt or terminal to complete the log in.
Once you’re logged in, it’s time to run the tests. On the last panel, you’d have seen a command to run:
appcenter test run uitest --app "Countr-iOS" --devices <your devices>--app-path pathToFile.ipa --test-series "master" --locale "en_US"
The <your devices> part will be a short hex string showing a unique identifier for the devices you selected, or the name used if you saved the device set. A couple of values in this command need to be replaced before you can run it:
- pathToFile.ipa or pathToFile.apk—This is the IPA or APK to upload and test against. If you downloaded the build from App Center, set this value to be the path to the downloaded build. If you built it locally, set this to the local package—for example, Countr.iOS/bin/iPhone/Debug/Countr.iOS.ipa would point to your Countr iOS debug IPA, or Countr.Droid/bin/Debug/io.jimbobbennett.Countr-Signed.apk for the Android version.
- pathToUITestBuildDir—This is the path to a folder containing the assemblies that contain the actual UI tests. This should be set to the output directory of the UI test project, so use Countr.UITestsinDebug.
Make these changes to the command, and then run it. It’ll take a few minutes to run, so while it’s running, let’s look in more detail at what we have run.
You may be asked for telemetry information
The first time you run an App Center command from the command line, you might be asked if you want to enable telemetry—analytics around how you use the CLI tools that Microsoft is gathering to help improve the user experience. Enter y to enable telemetry, n to disable it.
Breakdown of the command
Let’s take a look at the command and its parameters in more detail. The appcenter command is used to interact with all the available functionality in App Center. Table 15.1 shows the different parameters used to submit tests.
Table 15.1. The parameters for the App Center command
|
Part |
Description |
|---|---|
| test run uitest | Tells App Center you want to run some Xamarin UI tests. |
| --app "<app name>" | Sets the app name for the test you want to run. App names on their own refer to apps owned just by you. For an app owned by an organization, use the format "Organization name/app name"—for example, "Xamarin-In-Action/Countr-Android". The name should be in quotes because app names can have spaces in them. |
| --devices <your devices> | A unique ID defining what devices to use. If you saved your device set, it will be the name you set when saving (prefixed with your organization name if your app is part of an organization). Otherwise it will be a unique ID generated by Test Cloud that references your devices. |
| --app-path pathToFile.ipa/apk | The path to the IPA or APK for your app. If you used the one built by App Center, this is the path to wherever you downloaded it. If you built it locally, this will be under the current folder, so Countr.iOS/bin/iPhone/Debug/Countr.iOS.ipa to point to your Countr iOS debug IPA or Countr.Droid/bin/Release/io.jimbobbennett.Countr-Signed.apk for Android. |
| --test-series "master" | The series to run your test under. Tests can be grouped by series so you can compare different test runs or builds. For example, you could have a develop series for builds that you’re actively working on that just tests recent changes, and a release series for your release candidates that tests everything. |
| --locale "en_US" | The device language to use, which is useful if you want to test your app with multiple languages. Test Cloud only supports a limited set of languages at the moment, but this list should grow over time as Microsoft has always been superb at supporting multiple languages. |
| --build-dir pathToUITestBuildDir | The path to the folder containing the assemblies that have your UI tests. These need to be compiled assemblies, so make sure it points to an output path and that your tests are built (for example, Countr.UITestsinDebug). |
What happens when you run this command
When you run this command, it will start by uploading your IPA or APK and tests to Test Cloud. It will then validate that your app package will work on the devices you’ve selected (for example, checking that it supports the OS version on the device). Once that’s validated, it will wait for the devices to be ready. When one is ready, it will start the test on that device, and as the others become available, the tests will start on those devices. If you use a popular device, you could be waiting a while—Xamarin is adding more devices all the time, but there can still be a wait. Once the tests are run, you’ll see the pass and fail counts in the command-line window. Figure 15.15 shows this process.
Figure 15.15. The App Center CLI uploads your binaries, validates them, and then runs the tests on the selected devices.
How long this takes depends on the speed of your network connection (after all, you have to upload a multi-megabyte package) and the availability of the devices. My last run, for example, took 10 minutes. The output follows (with repeated lines replaced with ellipses to make the output much shorter):
Preparing tests... done. Validating arguments... done. Creating new test run... done. Validating application file... done. Uploading files... done. Starting test run... done. Test run id: "865a554c-5d0d-45d4-b748-5c15b8bec2ce" Accepted devices: - Apple iPad Air 2 (10.3.2) - Apple iPhone 7 (10.3.2) Current test status: Validating ... Current test status: Running on 2 devices (0 / 2 completed, 0 pending) ... Current test status: Running on 1 device (1 / 2 completed, 0 pending) ... Current test status: Tests completed. Processing data. Current test status: Done! Total scenarios: 3 3 passed 0 failed Total steps: 10
This shows that the three tests ran and passed. You’ll also receive an email from App Center with the test results.
Now let’s open up App Center to see more details.
15.3.5. Viewing the test results on App Center
To see the results of your tests, open up App Center for your app and select the Test tab. In there you’ll see a summary of all your test runs (figure 15.16).
Figure 15.16. The Test dashboard gives an overview of all your tests runs for your app.
This shows the number of tests over time and the peak memory usage of your app as line charts. Underneath these charts is a list of all the test runs in chronological order, with the most recent at the top. As you move your mouse over the lines in the charts, the corresponding test runs for that point will be highlighted. You can click the lines or the test runs in the list to see the details of an individual run.
Even though you’ve only run your tests once, you’ll see two tests showing: the test run you just did with three tests in it, and a run with a single test. The single test run was the launch test performed by your build, so not only can you tell from the build tab that it was successful, you can also see more details on it in here.
Click the test run you’ve just done to see a dashboard showing more details. This dashboard shows a summary of the run, with the date and time, the time taken, the version of your app, how long the test took, charts showing the success and failure breakdown by test and device, and a list of all the tests run. The list of tests includes the peak duration and memory usage, as well as the status. This is shown in figure 15.17.
Figure 15.17. The Test dashboard, giving an overview of the latest test run
You shouldn’t see any failures for your tests, but to see what failures look like, try breaking the app (just as we did back in chapter 14 to show UI test failures) and rerun the tests—not forgetting to revert the breaking code once the tests are done. Figure 15.18 shows what a test run with a failing test looks like.
Figure 15.18. The Test Cloud dashboard showing a failing test
When you click a test in the list, you’ll be shown a breakdown of that test (figure 15.19). The left panel will show a tree of all the tests in the run, with the test you clicked expanded. It will show a list of steps that match up to the calls to IApp.Screenshot() that you added to your UI tests back in chapter 14 to capture named screenshots at various points in the test. As you click each step, you’ll see the screenshot that was captured for each of the devices in your test run, and this screenshot will be updated as you click into the different steps. You can then click an individual device to see more details, such as the test duration, memory usage, and CPU usage at each screenshot. You can also access device information and even the system log (useful sometimes for debugging test failures). This is shown in figure 15.20.
Figure 15.19. The details of a test run
Figure 15.20. For each test you can see a screenshot from an individual device, as well as device metrics.
The ability to see screenshots can be an incredibly powerful tool. When you design your app, it’s hard to know how it will look on all devices. You can use simulators to test all available iOS devices, but this is time consuming. Testing all possible Android devices manually would take a huge amount of time and require a lot of configured emulators or a large pile of physical devices. By using Test Cloud inside App Center and capturing screenshots, you can build up UI tests that work through your app, capturing screenshot after screenshot from all the screens in your app. You can then use the App Center dashboard to review these screenshots and quickly compare multiple devices. This is a fast way of seeing how your app looks, and it’s a great way to justify the expense of App Center—the cost of paying developers to check apps on multiple devices is way higher than the cost of App Center!
You’ve now got your app building on iOS and Android and tested in the cloud. The next step in preparing for release is to set up analytics and crash reporting, so that once your beta test and production users have your app, you can not only analyze how the app is used, but also monitor for any crashes in order to fix them as soon as possible.
15.4. Analytics and crash reporting
When you finally release your app to your users, you want them to have the best experience possible. Good experiences lead to repeated use, good reviews, and recommendations; bad experiences lead to users deleting your app and giving bad reviews (with bad reviews leading to potential users skipping your app for a competitor’s). One way to guarantee a bad experience is to have an app that crashes. You can test your app to reduce the chance of crashes, but they’re a fact of life—users always seem to end up doing something developers didn’t expect, causing a crash. Even extensive unit and UI testing can’t cover everything.
You may not be able to prevent your app from crashing, but you can monitor for crashes, find out what causes them, and fix it as soon as possible. Your users may well forgive a crash if they get a fix pretty swiftly, and this is where crash monitoring comes in. App Center has an API you can add to your app that will monitor for crashes, and once the app is restarted it will upload a crash report so that you, as the developer, can see what caused the crash and fix it.
To help give users the best experience possible, you should also take time to understand your audience—to see what type of devices or OS versions they use, see what parts of the world they live in, and discover what languages they speak. You can use this information to tailor your app to give your users a five-star experience. App Center also has an API to help with this.
15.4.1. Adding the App Center SDKs
To enable crash reporting and analytics, the first thing you need to do is install the App Center NuGet packages, so head back to Visual Studio and load your app. Install Microsoft.AppCenter.Crashes and Microsoft.AppCenter.Analytics into the Countr.Core, Countr.Core.Tests, Countr.iOS, and Countr.Droid projects (figure 15.21).
Figure 15.21. Adding the App Center NuGet packages
Once the NuGet packages are added, configure the SDK inside each project. Part of this configuration is to connect the SDK with the particular app inside App Center, so that it knows where to send the crashes and analytics. This connection is configured using an app secret that’s generated for you when you create your app—this secret is a GUID that’s unique to your app and can be found by selecting Settings from the menu. On the Settings tab, you’ll see the app secret at the top right.
Setting up the SDK on Android
The SDK needs to be configured as early on in your app’s lifecycle as possible, so the easiest place to do this is inside the splash screen. This way everything is configured as soon as the app starts up, and any crashes that happen during startup can be caught.
Add the following code to the SplashScreen class in the Droid project.
Listing 15.1. Adding App Center to Countr.Droid
using Microsoft.AppCenter;
using Microsoft.AppCenter.Analytics;
using Microsoft.AppCenter.Crashes;
...
protected override void OnCreate(Android.OS.Bundle bundle) 1
{
base.OnCreate(bundle);
AppCenter.Start("<your app secret>", 2
typeof(Analytics), 2
typeof(Crashes)); 2
}
- 1 Adds an override for the OnCreate method
- 2 Starts App Center using your app’s secret
Replace <your app secret> with the value of the app secret for your Android app. Remember, the iOS and Android versions are different apps in App Center, and they have different app secrets, so make sure you use the right one. This line tells App Center to start both analytics and crash reporting.
App Center requires three permissions: ACCESS_NETWORK_STATE, INTERNET, and READ_EXTERNAL_STORAGE. The SDK will automatically add these permissions for you when you compile your app, so you don’t need to add them manually and you won’t see them ticked in the Android manifest file.
Setting up the SDK on iOS
On iOS, the best place to configure App Center is in AppDelegate, so add the following code to the FinishedLaunching method override, again replacing <your app secret> with the value of the app secret for your iOS app.
Listing 15.2. Adding App Center to Countr.iOS
using Microsoft.AppCenter;
using Microsoft.AppCenter.Analytics;
using Microsoft.AppCenter.Crashes;
...
public override bool FinishedLaunching(UIApplication app,
NSDictionary options)
{
AppCenter.Start("<your app secret>",
typeof(Analytics),
typeof(Crashes));
...
}
15.4.2. Understanding your audience
Audience data is invaluable in enabling you to make decisions about your app. If you have a large number of customers in certain countries using languages that your app is not available in, then perhaps it’s time to add translations for that language. If users are on old OS versions, you may need to perform extra testing or support. If you’ve targeted your UI toward phones, but most users are on tablets, maybe you need to rethink your UI. All this information can be found in App Center’s analytics.
Once you’ve added the App Center SDK to your apps, fire one of them up and play with it for a while. Then open App Center for the app you’ve been using and head to the Audience page under the Analytics tab in the menu. It make take a minute or two for this information to update, but when it does you’ll see a breakdown of the usage for your app.
At the top of this page are a couple of filters, allowing you to filter the information you see based on app versions or date ranges. The preview version of App Center only supports 90 days of data, but once it’s fully released you should be able to pay more to access more data if needed. From here you can see how many users use your app daily, how long they spend in your app, which versions of your app they use, their device types, OSs, languages, and locations.
- Active Users— This panel shows a graph with three lines, giving the number of unique devices that have used your app over the past day,
week (seven days), and month (30 days)—the day starts at midnight UTC, so it may not reflect a local “day” for you (figure 15.22). This panel can give you a great indication about the increase in app users, and by looking for sudden peaks, you can see
when larger numbers of people started using your app—maybe you can tie it back to a marketing push or a new app version. If
you see this number trailing off, maybe it’s time to add new features and do a marketing push to highlight these to your users,
to keep them coming back.
Figure 15.22. Active users shows how many unique users use your app each day.
- Engagement— It’s all well and good seeing that new users are signing up for your app, but you also need to consider how engaged they are.
Are they regularly coming back as opposed to using it once and deleting it, and are they spending enough time in your app
to get the most from it? The Engagement tab can give you this information (figure 15.23). The Daily Sessions Per User shows how many times per day your app is run on each unique device, displaying a graph with
a count against dates; the Session Duration shows how many users are using your app for a certain period of time, split into
buckets ranging from a few seconds to over an hour.
Figure 15.23. Engagement can show you how many times each user uses your app, and for how long.
- Devices— The Devices section shows the top four devices that your users have used, the number of users on each device type, and a percentage
breakdown (figure 15.24). It also shows the top four OS versions, and what percentage of the users on those versions are on each.
Figure 15.24. Devices gives a breakdown by device type and OS.
- Country/Region and Languages— This section gives a breakdown by country, showing a map identifying the countries your users are based in. Darker shading
of a country means a higher percentage of users. This country data is taken from the country of the mobile carrier, so it’s
only available on devices with a SIM card—most of the data will come from phones, with much less from tablets because a lot
of tablets don’t have cellular connectivity. This section also shows the name and percentage of users in the top seven countries.
The language data shows the top seven major language settings from the device. English would include British users, along
with New Zealand English, Australian English, and US English. This is shown in figure 15.25.
Figure 15.25. Breakdown by country and language
- Versions— The versions section shows the top 20 app versions that your users are using (figure 15.26).
Figure 15.26. Versions breaks down your app usage by app version.
15.4.3. Adding event tracking
Just by adding the App Center SDKs and calling the Start method, you get user analytics, and this is useful for understanding more about your users. The next thing you want to learn is what your users are doing with your app, what steps they take, or how often they use certain features. You can use this information to understand what users do most often and how they do it, and maybe this can help you improve your app. For example, in the Countr app, if users are deleting counters a lot and immediately adding new ones with the same names, they’re probably trying to reset the counter to 0. This would suggest you should add a “reset count” option.
As well as tracking what users do, you can also attach information to these events, so that you can learn more about their behavior. For example, in a news app you could track not only that a user wants to see the entire story behind a headline, but also what story they tap on, to determine what stories are more popular. In a camera app you could track not only that the user has shared a photo, but where they shared it to.
Monitoring and privacy
If you’re monitoring users, you need to consider their privacy. These analytics don’t capture any personally identifiable information by default, but you could easily add user-identifiable data, such as tracking a login with the username they’ve used. It’s good practice to have a privacy policy, so that users know that you’re capturing data, what data you’re capturing, and what you plan to do with it. Also check the laws in the countries in which you operate, as there may be legal considerations.
Simple event tracking
The simplest event tracking is to track a named event. In Countr, a good event to track would be the user tapping the plus button to add a new counter. The App Center SDK works in cross-platform code as well as in platform-specific app projects, so you can add the event tracking to your view model.
Open CountersViewModel in the Countr.Core project, and add the following code.
Listing 15.3. Tracking when the user shows the add new counter screen
using Microsoft.AppCenter.Analytics;
...
void ShowAddNewCounter()
{
Analytics.TrackEvent("Show add new counter"); 1
...
}
- 1 Tracks the event
You can use this to track an event with any name you want, within limits—you can only have 200 distinct event names in use, and each name must be less than 256 characters in length.
When you call TrackEvent, the data is queued up and sent in the background, usually pretty instantly. This means the thread isn’t blocked. If the device is offline, this data is persisted and sent the next time the device is connected to the internet. This data is also persisted between sessions, so if the device is offline and the app is terminated, the data will be sent the next time the app is run with the device online.
Adding data to events
Events can have properties attached as key-value pairs in a Dictionary<string, string>, allowing you to provide custom data to your events. Again, as with events, there are some limits to the data you can provide—each key and value is limited to 64 characters, and you can only add up to five properties to each event.
You can use this to track when the user saves a new counter, tracking not only the event but the name the user has assigned to the counter. This information could be useful. For example, if certain counter names are used a lot, you could offer that name as a predefined default somehow.
To add data to an event, add the following code to the CountersService in the Countr.Core project.
Listing 15.4. Tracking when the user adds a new counter and the counter’s name
using Microsoft.AppCenter.Analytics;
...
public async Task<Counter> AddNewCounter(string name)
{
...
var props = new Dictionary<string, string>(); 1
props.Add("Counter Name", name); 1
Analytics.TrackEvent("Add new counter", props); 2
return counter;
}
- 1 Creates the properties dictionary
- 2 Tracks the event with the properties
You can repeat this same pattern to track when the user deletes or increments a counter.
Seeing the events
Once you’ve started collecting data, you can see it in App Center in the Events page under the Analytics tab. Again, like analytics, you can filter based on time periods or particular app versions (figure 15.27).
Figure 15.27. App Center shows you all events, including the total count and for how many users this event was tracked.
For each event, you can see how many times it was tracked (including a delta since the previous time period), for how many users it was tracked, and an average number of times the event was tracked per user and per session. This gives a great overview of the popularity of a feature. You might hope that for Countr you’d see an increment counter event tracked for most of your users for each session, meaning that the app was being used on a regular basis for its intended purpose of counting things.
You can click an event to drill down into more information. When you do, it shows a breakdown over time of that event:
- Counts— Three panels provide the counts for the events—how many times the event was raised (figure 15.28). You can view a graph that shows on a day-by-day basis how many users executed the code that raised this event and the total
number of times the event happened. You can also see the average count per user and per session.
Figure 15.28. You can see the number of users who raised an event, and the total number of times the event was raised.
- Event properties— For events with properties, App Center will show all the properties that have been sent against this event (remember, you
can only send five unique properties against each named event). For each of these properties, it will show the top 10 values
with a count of how many times the event was raised with that value, and the change in that count over time. This is shown
in figure 15.29.
Figure 15.29. For events with properties, you can see the number of times the event was raised with specific values.
15.4.4. Crash reporting
App Center is able to provide crash reporting for your app. If your app crashes, the SDK can detect details about the crash and upload this information to App Center the next time the app is started. Once you have a crash report, you can fix the bug that caused it and hopefully get a fix to your customers before they uninstall your app and leave a bad review.
The App Center SDK does this by hooking into the .NET unhandled exception handler, as well as the OS-specific unhandled exception mechanisms. This means any exception that’s unhandled, be it in .NET code or native code, will be caught and sent up to App Center. When a crash does happen, there’s no guarantee that anything in your app will actually be working for the remainder of its life, so the exception details will be saved locally, and when your app is restarted the crash data will be sent.
Not all exceptions will be caught
App Center can catch most exceptions, but not all. Out-of-memory exceptions won’t be caught because there’s no memory left to catch them with. Also, due to a limitation with the Xamarin runtimes, stack overflow exceptions won’t be caught.
To see this in action, you can create a fake crash and see the report in App Center. To create an exception, you can add a throw statement to throw a new exception. Add the following code to the CountersService in the Countr.Core project.
Listing 15.5. Crashing your app when you increment a counter
public Task IncrementCounter(Counter counter)
{
throw new System.Exception("Crash"); 1
...
}
- 1 Throws an exception
Make this change and then run the app. When you increment a counter, it will crash. Relaunch the app so that the crash report can be uploaded, and then head to App Center and open the Crashes tab (figure 15.30). As with the analytics and events, you can filter this data by date range or version.
Figure 15.30. The Crashes tab shows crash-free users per day and a list of the crashes received.
The first part of this tab shows two graphs. The first shows the percentage of users per day that are crash-free—those who managed to use your app for the day without experiencing any crashes. The second shows a total count of crashes each day in the selected time period. This allows you to make a decision about how important a crash fix is—if you have 100 users and 50 of them get a crash each day, then it’s pretty important. But if you have millions of users and only one or two get a crash, maybe it can wait for your next scheduled release.
The second part shows a list of crash groups—groups of crashes that are the same. For example, if in one version of your app 100 users got the same crash in the same place, you wouldn’t see 100 rows here. Instead you’d see one row for that crash showing it happened to 100 users. This makes it easier to manage crashes.
If you click a crash group, you can see more details of the crash (figure 15.31). You can see details of the affected users, including how many users got the crash and how many times, and which devices it occurred on. This is useful information. For example, you might only get a crash on a particular OS version or a particular device, meaning you need to source a device or simulator matching those characteristics for testing your fix.
Figure 15.31. Crash groups show details on affected users and the stack trace of the crash.
The stack trace gives you a breakdown of where the crash happened. In the example in figure 15.31, you can see it happened in IncrementCounter, which is expected, as this is where you put the throw. The stack trace looks a little weird, with the exception actually happening in CountersService+<IncrementCounter>d__6.MoveNext (), a method that doesn’t exist. This is one of the downsides to async/await—the compiler creates some magic behind the scenes to handle the thread-switching, and this appears in the stack trace. You have to look at the created method names and look for things that match your code, such as the IncrementCounter buried inside the compiler-created method name.
The default stack trace tries to only show you relevant code, and it highlights what it thinks are methods from your code, not the SDKs. You can expand the stack trace to see the full trace, including all SDK methods, if you need to.
The stack trace information is created using symbol information—data that links the calls inside your binary to the source code. For Android, this is stored inside the app package, but for iOS this isn’t available. This means that you’ll need to provide a symbol file in order to see a stack trace on iOS. If you built your app using App Center, it will automatically get the symbol file from the build, but if you built locally you may need to provide the symbol file. The Symbols tab under Crashes will show you if there are crashes with missing symbols, and from there you can upload your symbol files. These files are created at the same time as the IPA when you compile your app, and they’re output to the same place and have the file extension .dsym.
Crash groups have a status—Open, Closed, or Ignored. All new crashes default to Open, and you can change the status as needed. Once you fix a crash, you can mark it as Closed to show it’s no longer an issue and can be ignored. You can also mark crashes as Ignored if needed. For example, if you’re developing something and get a crash as part of your development work, and you fix it before the code is released, you can mark that crash group as Ignored. You can filter crashes on the dashboard based on their status, and the filter defaults to Open, only showing open crashes.
Revert the throw!
Once you’ve played with the crash reports, don’t forget to remove the throw!
You’ve built your app, tested it on real devices in the cloud, and set it up to track your users and report crashes. You’re finally ready to put it in the hands of your users. In the next chapter we’ll look at using App Center to distribute your app to beta test users, before finally uploading it to the store.
Summary
In this chapter you learned
- App Center is “Mission Control for apps” and provides services for the mobile-optimized development lifecycle.
- You can use App Center to build and test your apps as well as viewing analytics, user events, and crash reports.
You also learned how to
- Set up apps and builds in App Center.
- Run UI tests on real devices in the cloud using Test Cloud from inside App Center.
- Add the SDKs to your apps to capture analytics, user events, and crash reports.
- View analytics on your apps and see what steps users are taking inside your apps.
- See crash reports and use them to manage app crashes.