Chapter 13. Testing applications with Spectron

This chapter covers

• Using Spectron to test Electron applications
• Understanding the relationship among Spectron, WebdriverIO, and Selenium WebDriver
• Controlling Electron APIs within integration tests

At this point in the book, we’ve built a number of Electron applications. Testing these applications to make sure that they work as expected has been something of a manual process. Make a change, start the application, and click around to ensure that everything works. This is fine for small applications, but this method can become tedious in larger applications. What if a change in the code in one part of the application breaks some functionality elsewhere?

In these situations, it’s helpful to automate the tests. Let the computer do the boring, repetitive work, while you focus on the features that bring value to your customers. That sounds great, but how do we do that? In a traditional web application, you might point something like Selenium WebDriver to your website, make some assertions as to how the site should behave, and then have it click around and confirm that everything works as expected.

That’s great, but we have a few problems: our Electron applications don’t have web addresses per se, but they do support APIs that give us unprecedented access to the underlying operating systems, which are typically not supported by web browsers. At this point, it shouldn’t surprise you that this tricky problem has an easy solution. The Electron team supports a project called Spectron, which allows developers to write integration tests for our Electron applications.

In this chapter, we write integration tests for a special version of Clipmaster 9000 that does not live in the menu bar, as shown in figure 13.1. We test all the core functionality and learn a few tricks along the way. To get started, you need to add Spectron and a test runner to your package.json. I’ve opted to use Mocha in an attempt to stay consistent with the official documentation, but you could use Jest, Karma, Jasmine, or any other test runner that makes you happy. I’ve included this in the example repository (http://mng.bz/UxHY), but if you’re working on your own, you can run npm install --save-dev spectron mocha.

Figure 13.1. A simplified version of Clipmaster 9000 that does not live in the menu bar.

First, it’s important to understand the relationship among Spectron, WebdriverIO, and Selenium WebDriver because we use methods from more than one of them in a given test. Let’s start at the bottom and work our way up. Selenium WebDriver allows developers to write tests that control a web browser so they can test their applications from a user’s perspective—as opposed to unit tests, which exercise a given piece of code in isolation. Selenium WebDriver is a language-agnostic library and is typically wrapped by a library that gives it native API bindings for a given programming language.

Enter WebdriverIO, which wraps Selenium WebDriver with a pleasant JavaScript API and makes it easy to use from within Node.js—or an Electron application, in our case. But as I pointed out in the beginning of this chapter, Electron applications have some major differences from traditional web applications, as well as a lot of additional power and functionality. Spectron wraps WebdriverIO, as shown in figure 13.2, and allows us to access this functionality in an environment custom-tailored for Electron applications.

Figure 13.2. Spectron is a wrapper around WebdriverIO, which—in turn—is a wrapper around Selenium WebDriver.

We won’t need to touch Selenium WebDriver in this chapter. WebdriverIO takes care of controlling Selenium WebDriver on our behalf. We douse methods from both Spectron and WebdriverIO. In practice, you don’t really have to think about the line between the two, but some of Spectron’s methods delegate to WebdriverIO. Where this ends up being important is when it comes time to look up the documentation for a given method. I’ll make sure to point out when we’re delegating to WebdriverIO, as opposed to using a method that belongs to Spectron itself.

13.1. Introducing Spectron

Spectron makes it easy to start our application and control its UI from our test suite. The primary way to use Spectron is to create an Application instance. This object includes a number of child objects that allow us to access different parts of our application, as shown in figure 13.2

Figure 13.3. The Spectron API

Spectron is broken up into the following parts:

1. client is the underlying WebdriverIO instance and exposes all of its methods. (This is only partly true, as I explain in the next section.) You use this when you want to search the DOM for a particular node or trigger click events.
2. electron is Electron’s renderer process API. Anything available when using require(’electron’) in the renderer process is available here. As a result, you can use electron.remote to access the main process in the same manner as you would in the renderer process.
3. browserWindow is a convenient alias to access the currently focused browser window in your application. It’s equivalent to electron.remote.getCurrent-Window(). In addition, browserWindow.capturePage() is useful when you want to take a screenshot of the currently active browser window and save it to the desktop. This can be useful when you’re trying to diagnose why your tests are failing.
4. webContents is an alias to Electron’s webContents API, which is useful for getting information about, or controlling the browser that is executing, your application. It is an alias for electron.remote.getCurrentWebContents(). Throughout this book, we’ve used this API to load an HTML page into a newly created BrowserWindow instance or to toggle the developer tools. The API provides access to browser functionality like the forward and back buttons, printing the page, setting the zoom level, and reloading the page. It emits events as the browser loads that can be useful for testing. It also provides the webContents.savePage() method, which allows you to save the currently loaded page as an HTML file on your filesystem for further inspection if your tests are not behaving as expected.
5. mainProcess is an alias to Node’s process global in the main process. It is equivalent to electron.remote.process. It’s important to note that main-Process is not an alias to Electron’s main process API, but it can be accessed using electron.remote. mainProcess is useful if you need to access the environment variables or the arguments passed to Electron when it was started.
6. rendererProcess is similar to mainProcess and provides access to the renderer process’s global.process object. Again, this can be useful for reading environment variables. If you need Electron’s renderer process APIs, use electron instead.

The first two properties of the Application instance provide deep access to the browser’s APIs as well as Electron’s own APIs. With this combination, we have deep, programmatic insight into and control over almost all of the inner workings of our application at any time. This access makes testing Electron applications easy once you get the hang of it. The browserWindow, webContents, mainProcess, and rendererProcess properties provide convenient aliases to the first two properties.

13.2. Getting comfortable with Spectron and WebdriverIO

In the previous section, I mentioned that I was fibbing I bit when I said that the client property delegated to WebdriverIO. That statement is technically true—but it isn’t the whole truth. client also has additional, Electron-specific methods, shown in figure 13.4, to make your testing experience even more pleasant.

Figure 13.4. Spectron adds a number of utility methods to the WebdriverIO API. This is an exploded view of the client in figure 13.3.

Some of the methods look familiar if you’ve written integration tests in the past. Methods like getSelectedText(), for example, consider that your application might have multiple windows—something that isn’t possible in a traditional web application. Others are unique to testing Electron applications. I introduce you to these methods now as a reference. We use many of them in the sections that follow.

getMainProcessLogs() and getRenderProcessLogs() return a promise that resolves to an array of messages that have been logged to the respective console. As soon as either method is called, it clears the messages from the console. This means that subsequent calls contain only messages that have been logged to the console since the last time the method was called.

getWindowCount() returns a promise that resolves to an integer that—unsurprisingly—represents the number of windows currently open in the application. This value might be useful if you are writing tests for an application like Fire Sale, which has a button allowing users to open another window. app.browserWindow references the currently focused window. app.client.windowByIndex() allows you to focus on an alternate window instead.

waituUntilWindowLoaded() takes a given number of milliseconds as an argument and returns a promise. If the window loads in the allotted time, the promise resolves. Otherwise the promise is rejected. This method allows you to delay execution of your tests until you’ve confirmed that the window has actually loaded.

waitUntilTextExists() takes three arguments: a selector, a string of text, and an optional number of milliseconds to wait before giving up. It returns a promise that resolves when the selector contains the text, Alternatively, it rejects if the 5 seconds or the provided number of milliseconds pass and the text has not appeared. This setting is useful for UI components where the content may come in asynchronously shortly after the window has loaded. Normal tests would fail because they would run immediately, before the asynchronous content has loaded.

13.3. Setting up Spectron and the test runner

Spectron provides the ability to start an Electron application and control it from your tests, but it doesn’t provide a framework for running your tests or verifying that your application works as expected. As mentioned earlier, we use a simplified version of Clipmaster 9000. A starting point for the code can be found here at https://github.com/electron-in-action/clipmaster-9000-spectron.

This is a very good thing because it allows you to use whatever test runner and assertion library you prefer. Just require Spectron in your test runner of choice, configuring it as we’ll see in listing 13.2. In this chapter, I use Mocha along with Node’s built-in assert library, but you can certainly use Chai and an assertion library or Jest, Karma, Jasmine, or any other framework, if you prefer. Spectron is not opinionated on this front. Instead, it allows you to drive your Electron application from the framework of your choice.

I’ve already included Mocha as one of the development dependencies for the example project. As I mentioned earlier, if you’re working on your own application and following along, you can run npm install --save-dev spectron mocha from the command line to install both Spectron and Mocha. This will install the mocha command line tool into your node_modules directory, which can be run by typing ./node_modules/.bin/mocha. This can be tedious. It would be much easier to assign it to be run using npm test. So, let’s add it to our project’s scripts in our package.json. (This should already be set up for you if you’ve cloned the repository from GitHub.)

Listing 13.1. Setting up Mocha as a test script: ./package.json

  "scripts": {
    "start": "electron .",
    "test": "mocha"           1
  }

• 1 Sets npm test to run the locally installed version of Mocha

Now whenever we run npm test, Jest runs our test suite by looking for all files with a *.test.js suffix. You can also use this suffix for any unit tests that you want to write—Mocha runs those as well. In a larger application, you might create a folder for all of your tests or include them alongside your implementation files. For the sake of simplicity, I’m going to use a single file called test.js to hold all of the tests for our application.

Let’s begin by examining the boilerplate provided in the repository to start our application before each test and stop the application after the individual test has run. This gives us a fresh instance of the application for each test, which eliminates the need to clean up any of the side effects and leftover state from previous tests.

Listing 13.2. Writing up Spectron with the test runner: ./test/spec.js

const assert = require('assert');                      1
const path = require('path');                          2
const Application = require('spectron').Application;   3
const electronPath = require('electron');              4

const app = new Application({
  path: electronPath,                                  5
  args: [path.join(__dirname, '..')],                  6
});

describe('Clipmaster 9000', function () {
  this.timeout(10000);                                 7

  beforeEach(() => {
    return app.start();                                8
  });

  afterEach(() => {
    if (app && app.isRunning()) {
      return app.stop();                               9
    }
  });
});

• 1 Requires Node’s built-in assertion library
• 2 Brings in Node’s helper utility for working with file paths
• 3 Pulls in Spectron’s application driver
• 4 Requires Electron. This will give us access to a locally installed development version of Electron.
• 5 Tells Spectron’s application to use the locally installed development version of Electron
• 6 Points to the root directory of the application itself as a starting point for the application
• 7 Increases Mocha’s default timeout because launching the application can take a while
• 8 Starts the application before each test
• 9 Stops the application after each test

The first thing we need is the location of the Electron application that we want to test. Theoretically, you could point Spectron to any of the Electron-based applications on your computer and control them programmatically, but I leave that as a devious exercise for the reader. I’m going to be a boring rule follower and get the path to the locally installed version of Electron that we pulled down from npm when we installed our dependencies using npm install.

We want access to a variable holding the instance of our application in each of our tests, so we declare it in the describe() block. Before each test, we create an application instance and assign it to the app variable. The new Application() constructor takes the path of the Electron binary, as well as the location of the directory where we would normally type npm start. This should be the directory that contains the package.json, which references the JavaScript file containing the code for the main process. After the instance has been created, we start the application. We’re now ready to test—despite the fact that we haven’t written a test case just yet. But imagine that we had a test and it has run. We check that there is a value assigned to the app variable and the application is still running. (It may have crashed.) If both of those cases are true, then we stop the application and move onto the next test case, where we go through this process again.

13.4. Writing asynchronous tests using Spectron

We’ve talked about our testing tools. We’ve set them up. Now let’s sit down and write some tests. Over the course of the chapter, we’ll check that Clipmaster

• Shows an initial window when it starts
• Displays “Clipmaster 9000” as the title
• Doesn’t have the Developer Tools open when it starts
• Has a button with the text Copy from Clipboard
• Doesn’t have any clippings displayed in the UI when the application starts
• Contains one clipping when the Copy from Clipboard button has been pressed a single time
• Successfully removes a clipping when a clipping’s Remove button has been clicked
• Has the correct text when a new clipping is created
• Writes the text of the clipping to the clipboard when the Copy button is clicked

We’ll start with a simple test: start the application, and confirm that this special version of Clipmaster 9000 creates a single browser window. Spectron and WebdriverIO use an asynchronous, promise-based API for controlling Electron and the web application within. Most methods return promises, which can be chained. As I’ve mentioned repeatedly throughout this book, Electron provides us with a modern version of Node and Chromium. We frequently have the latest and greatest browser and language features at our disposal. This includes the async/await syntax for working with asynchronous APIs in a way that’s as easy to wrap our head around as synchronous code. With this syntax, you don’t have to worry about passing around callbacks or chaining promises.

As shown in listing 13.3, we use the async keyword before the arrow function to denote that we’ll be using the async/await syntax in this function. The await keyword pauses until app.client.getWindowCount() has resolved and then assigns the result to the count variable. After this is done, we expect the count to equal one. Mocha expects asynchronous tests to return a promise when they’re done, so you need to make sure that you prefix your expectation with the return keyword. Your author failed to do that on multiple occasions when writing this chapter before his first cup of coffee. He paid dearly for that oversight with his time and sanity.

Listing 13.3. Writing a test to count the number of windows: ./test/spec.js

it('shows an initial window', async () => {
  const count = await app.client.getWindowCount();     1
  return assert.equal(count, 1);                       2
});

• 1 The application has started; get a count of all of the windows.
• 2 Verify that this version of Clipmaster creates only one window.

This test is simple, and a traditional implementation using promises would not look that much different, but more involved tests require longer promise chains that might become tedious to write or confusing to reason about at best.

13.4.1. Waiting for the window to load

In many tests, you have to wait for the window to load the HTML, CSS, and JavaScript before you can continue testing the application. Electron loads the HTML quickly, but it’s not instantaneous. We frequently use the app.client.waitUntilWindowLoaded() method described earlier to wait until the application has fully loaded before moving forward. Let’s write a test confirming that the window’s title is the name of the application.

Listing 13.4. Writing a test to verify the window title: ./test/spec.js

it('has the correct title', async () => {
  const title = await app.client.waitUntilWindowLoaded().getTitle();    1
  return assert.equal(title, 'Clipmaster 9000');                        2
});

• 1 After the window has loaded, gets the title of that window
• 2 Verifies that the title of the window is what we expect

Many of Spectron’s methods are chainable, with each subsequent method in the chain being called after the promise returned from the previous method has resolved. In the previous example, we wait for the window to finish loading and then get the title of the window and store it in the title variable; after that asynchronous operation is complete, we can verify that the window’s title is what we expect it to be.

13.4.2. Testing Electron BrowserWindow APIs

Traditionally, integration-testing tools like Selenium and WebdriverIO point a browser to a webpage and interact with it. They typically don’t have much more access to the internals of the browser itself like we do as developers. Electron allows developers to programmatically open the developer tools when the window first loads. That said, it would be bad if we accidentally forgot to remove this code and shipped this to users. Let’s write a test that verifies that developer tools are not opened when the application is loaded.

Listing 13.5. Testing that the developer tools are not open: ./test/spec.js

it('does not have the developer tools open', async () => {
  const devToolsAreOpen = await app.client
    .waitUntilWindowLoaded()                           1
    .browserWindow.isDevToolsOpened();                 2
  return assert.equal(devToolsAreOpen, false);         3
});

• 1 Gets a reference to the browser window instance
• 2 Checks to see if the developer tools are open
• 3 Verifies that the developer tools are not open

The isDevToolsOpened() method is available on all BrowserWindow instances. When the application has loaded, we get the current browser window and ask it whether it has the developer tools open. We expect this to be false. If it is true, the test will fail.

13.4.3. Traversing and testing the DOM with Spectron

In addition to testing Electron APIs, we almost definitely want to test the UI of our application like we would a traditional application. Let’s start with a simple example where we search the page for a particular element and verify that it has the copy that we expect.

Listing 13.6. Testing the content of the Copy from Clipboard button: ./test/spec.js

it('has a button with the text "Copy from Clipboard"', async () => {
  const buttonText = await app.client
  .getText('#copy-from-clipboard');                           1
  return assert.equal(buttonText, 'Copy from Clipboard');
});

• 1 Uses the WebdriverIO API to get the text from a DOM node on the page

The getText() method is provided by WebdriverIO as opposed to Spectron. It accepts a selector as an argument, finds the first node that matches that selector, and returns a promise that resolves to the text content of that node. Finally, we confirm that the text is what we expect it to be.

This works, but what about more complicated traversal? When the application starts, the clippings list is empty—remember this was before we knew how to persist data in our Electron applications. Each time the user clicks that Copy from Clipboard button, a new clipping should be added to the page. How might we go about testing these scenarios? Let’s start with the first case: when the application starts, there should not be any clippings on the page.

Listing 13.7. Testing the initial state of the UI: ./test/spec.js

it('should not have clippings when it starts up', async () => {
  await app.client.waitUntilWindowLoaded();                        1
  const clippings = await app.client.$$('.clippings-list-item');   2
return assert.equal(clippings.length, 0);                          3
});

• 1 Waits until the window has loaded its content
• 2 Uses WebdriverIO’s API to find all the clipping list items on the page
• 3 Verifies that, by default, there are no clipping list items

Webdriver exposes $ and $$ methods, which are aliased to document.querySelector and document.querySelectorAll, respectively. We use $$ to select all of the clipping list items on the page, which returns a NodeList object. Ideally, there are no clipping list items on the page and the length of that NodeList is 0.

So far we’ve inspected and verified the state of the application when it starts—an important task but typically only a small portion of the tests we write for a given application. We’ve confirmed that there are no clippings on the page when the application starts. It stands to reason that when the user adds a clipping, it should be added to the UI. We already know how to check for the number of elements matching a given selector on a page. But how do you add a clipping? You click the Copy from Clipboard button. We need to figure out how to program WebdriverIO to click on a particular element on a page.

Listing 13.8. Testing a click interaction: ./test/spec.js

  it('should have one clipping when the "Copy from Clipboard" button has been
     pressed', async () => {
    await app.client.waitUntilWindowLoaded();
    await app.client.click('#copy-from-clipboard');                  1
    const clippings = await app.client.$$('.clippings-list-item');   2
  return assert.equal(clippings.length, 1);                          3
  });

• 1 Triggers a click event on the Copy from Clipboard button
• 2 Finds all of the clippings on the page
• 3 Verifies that there is one clipping now

Lucky for us, app.client provides a click() method that does exactly what you might expect: it triggers a click event on a node that matches the selector provided. When we click the Copy from Clipboard button, we expect that a clipping is added to the page. The prior test verifies that there is one clipping on the page after the Copy from Clipboard button has been clicked.

What about the flip side? If you recall, clippings can also be removed from the UI. But there is a catch—the Remove button appears only when the user’s cursor is hovering over the clipping. We can’t simply query for the Remove button, because it’s not on the page yet. Instead, we need to take control of the cursor and move it over the clipping to have the Remove button appear.

Listing 13.9. Moving the cursor in a test: ./test/spec.js

  it('should successfully remove a clipping', async () => {
    await app.client.waitUntilWindowLoaded();
    await app.client
      .click('#copy-from-clipboard')
      .moveToObject('.clippings-list-item')                        1
      .click('.remove-clipping');                                  2
    const clippings = await app.client.$$('.clippings-list-item');
  return assert.equal(clippings.length, 0);                        3
  });

• 1 Moves the cursor to the clipping to trigger the Remove button to appear
• 2 Clicks the Remove button
• 3 Verifies that the clipping is no longer on the page

After clicking the Copy from Clipboard button, we use the moveToObject() method to move the cursor over the new clipping, which causes the Remove button to appear. We click Remove and then verify that the clipping is no longer on the page.

const app = new Application({
  path: electronPath,
  args: [path.join(__dirname, '..')],
  webdriverOptions: {
    deprecationWarnings: false
  }
});

13.4.4. Controlling Electron’s APIs with Spectron

At this point, we have tested that we can add and remove clippings from the page, but we still have a major blind spot: we have no idea whether the clipping is displaying the correct text. If you were able to squint at the application during the previous tests, you might have noticed that the clipping that was added contained whatever was on our clipboard at the time. This will be tricky because our tests should be isolated from the outside world.

What we need to do is use Electron’s clipboard module to write text to the clipboard and then verify that the provided text is, in fact, what is displayed in the UI. Luckily, Spectron gives us access to all of Electron’s APIs. This means we can manipulate what is currently on the clipboard as part of the test and confirm that it is what’s found in the UI when the clipping is added to the page.

Listing 13.10. Accessing Electron APIs in a test: ./test/spec.js

  it('should have the correct text in a new clipping', async () => {
    await app.client.waitUntilWindowLoaded();
    await app.electron.clipboard.writeText('Vegan Ham');               1
    await app.client.click('#copy-from-clipboard');
    const clippingText = await app.client.getText('.clipping-text');   2
  return assert.equal(clippingText, 'Vegan Ham');                      3
  });

• 1 Accesses Electron’s API to write text to the system’s clipboard
• 2 Gets the text of the new clipping that was created
• 3 Verifies that the new clipping contains the content of the clipboard

The clipboard module is available in both the main and renderer processes, so there is no need to use electron.remote. Before clicking the Copy from Clipboard button, we write text to the clipboard. We then click the button and verify that the clipping’s text is exactly what we wrote to the clipboard.

This covers half of Clipmaster 9000’s functionality. The flip side of the coin is that the user should be able to copy the clipping back to the clipboard when the clipping’s → Clipboard button is clicked. To test this, we need to build on the work we did in the previous test. After clicking the Copy from Clipboard button and adding it to the page, we change the clipboard’s content to something else, then click the → Clipboard button, read from the clipboard, and verify that it is back to the original text that we added to Clipmaster 9000.

Listing 13.11. Testing that the application writes to the clipboard: ./test/spec.js

  it('should write the clipping text to the clipboard', async () => {
    await app.client.waitUntilWindowLoaded();
    await app.electron.clipboard.writeText('Vegan Ham');                1
    await app.client.click('#copy-from-clipboard');                     2
    await app.electron.clipboard.writeText('Something different');      3
    await app.client.click('.copy-clipping');                           4
    const clipboardText = await app.electron.clipboard.readText();      5
  return assert.equal(clipboardText, 'Vegan Ham');                      6
  });

• 1 Writes text to the clipboard using Electron’s API
• 2 Clicks the Copy from Clipboard button
• 3 Writes some other text to the clipboard
• 4 Clicks the button that should allegedly copy the text back to the clipboard
• 5 Reads the text from the clipboard
• 6 Verifies that the text is now the content of the clipping and not the text we manually wrote to the clipboard in our test

With this final test in place, we can verify that all of the UI elements work and behave as expected. We were able to use a combination of WebdriverIO and Electron APIs to test the application from multiple angles. The code as it stands at the end of this chapter can be found on the completed-example branch of the repository.

Summary

• Spectron is an officially supported library for testing Electron applications.
• Spectron wraps WebdriverIO, which provides Selenium with Node.js bindings.
• All of Electron’s APIs are available in Spectron.
• Spectron does not provide its own test running or assertion library. Instead, it allows you to choose which one you want to use.
• Electron supports async/await syntax, which greatly simplifies writing asynchronous code.