Now that we have covered building an in-house Selenium Grid and using a third-party provider, let's discuss some of the key benefits and limitations of both:
- Number of platforms: The major advantage of using a provider like Sauce Labs is that there is virtually an unlimited pool of browser and mobile platforms available. There is really no way to build an in-house grid with 900 browser/OS combinations or hundreds of mobile iOS and Android devices to test against. Along with all the different combinations, users can also test different browser versions, OS versions, mobile device versions, mobile device platforms, mobile device API versions, and so on.
- Maintenance: To go along with building an in-house grid, there is maintenance. Users have to keep up with installing OS patches, security patches, browser upgrades, Selenium upgrades, driver upgrades, and so on. On the mobile side, the Xcode or Android SDK and simulator/emulators have to be constantly upgraded, along with Appium, NPM, Node.js, and so on. The maintenance of supporting an in-house grid is one of the most time-consuming costs to have to deal with, whereas there is no maintenance if a third party is used, just a financial cost.
- Performance: The Sauce Labs Test Cloud is much slower than an in-house grid, by 25–30%. So, that would be one advantage of building an in-house grid. But, Sauce Labs has a paradigm that if you build tests small and modular, and design them to run in parallel, you can leverage multiple VMs simultaneously and actually get the tests to run faster. So, although Sauce Labs would introduce latency into the test run, it can be overcome if the tests are designed in a particular fashion. Also, when the Sauce Labs job is running, it is recording the session, creating logs from Selenium and, updating the dashboard with real-time results, which contributes to the latency.
- High-availability: If using an in-house grid, it's likely that software updates will be pushed out to VMs on the grid on a regular basis, which in effect, reboots the nodes. Services can be set up to restart the Selenium servers on the hub and nodes, but the forced reboots can make the availability of the nodes on the in-house grid sketchy at times. When using a third-party provider, it's likely the service will be up most of the time, or at least 99% of the time.
- Upgrades: Sauce Labs is continuously providing upgrades to new platforms, browsers, and mobile devices, both real and simulated. They support the latest Selenium, Appium, and browser driver revisions as they become available. For in-house grid upgrades, users have to schedule downtime in-between releases to bring down the hub and nodes, and those are done less frequently than Sauce Labs would provide.
- Enhancements: Sauce Labs supports Selenium-based plugins like BrowserMob Proxy for instance, and has the latest technologies tested and available for use. That would also include the new TestObject Real Device Cloud service they recently introduced to the market to support mobile testing on physical devices.