Selenium has been developed to test web applications. Selenium allows writing tests without the need to learn any test script language (Selenium IDE). In addition, C #, Groovy, Java, Perl, PHP, Python, Ruby and Scala, such as a series of popular programming language to provide testing environment. These tests can then be run on most modern web browsers. Selenium is an open source software under the Apache 2.0 license.

Selenium is designed to automate the operations of a web browser. With Selenium, any user can manually perform interactions that can be performed manually. Selenium can be used for any kind of automation, but the priority is to create automated web application tests.

The name of Selenium comes from a joke. Another automated testing framework was used as a popup during the development of Selenium, and the company was Mercury Interactive. Since Selenium is a well-known antidote to Mercury poisoning, this name has been suggested.

Selenium can be defined as an umbrella that encompasses a variety of tools and libraries that provide web browser automation. Selenium provides the infrastructure for the W3C WebDriver specification and provides a platform interface compatible with all major web browsers. The source code for Selenium is available with an Apache 2.0 license.

Selenium is intended for the automated testing of web applications, and it performs brilliantly as a general- purpose browser automation tool.

Selenium is a package designed to be used to test open source web applications for web applications for different browsers and platforms. The main purpose of Selenium is to automate web-based applications.

Web scraping is an important skill for data scientists. Each data set you want to analyze may not be available in a suitable format, and if you want to make unique analyzes, it is also important to make a decent data set by scraping yourself.

Static scraping is sufficient for retrieving data from static lists, but we need to automate the browser and interact with the DOM to retrieve data from a web site controlled by JavaScript, or to retrieve data from a parcel that is placed as an iframe item by JavaScript. One of the best tools for this purpose is Selenium

In this chapter, we will learn about the following topics:

• The advantages and disadvantages of using Selenium for web scraping
• RSelenium
• Step-by-step web scraping with RSelenium