In this section, we will perform a search on http://automationpractice.com to obtain a list of products that match the search query, illustrating the use of selenium.webdriver. Web elements are elements that are listed in a web page or that are found in a page source. We also look at a class called WebElement, used as selenium.webdriver.remote.webelement.WebElement.

To begin with, let's import webdriver from selenium, set a path to chromedriver.exe, create an object of webdriver—that is, driver, as implemented in the previous section, Accessing browser properties—and load the URL, http://automationpractice.com:

driver.get('http://automationpractice.com')

The new Google Chrome window will be loaded with the URL provided. Find the search (input) box just above Cart, as shown in the following screenshot: 

To continue searching through the script, we need to identify the element with the HTML <input>. Please refer to the Using web browser developer tools for accessing web content section in Chapter 3, Using LXML, XPath, and CSS Selectors.

In our case, the search box can be identified by the attributes shown in the preceding screenshot, or even by using the XPath or CSS selectors:

• id="search_query_top"
• name="search_query"
• class="search_query" 

The selenium.webdriver provides lots of locators (methods that are used to locate elements) that can be applied conveniently as applicable to the cases encountered.

Locators return single, multiple, or lists of WebElement instances, written as selenium.webdriver.remote.webelement.WebElement. The following are a few locators, along with a brief description:

• find_element_by_id(): This finds an element by its id attribute. This method returns a single WebElement.
• find_element_by_name(): This finds a single element by its name attribute. Multiple WebElements can be found or located using find_elements_by_name().
• find_element_by_tag_name(): This finds a single element by the name of its HTML tag. Multiple WebElements can be located using find_elements_by_tag_name().
• find_element_by_class_name(): This finds a single element by its class attribute. Multiple WebElements can be located using find_elements_by_class_name(). 
• find_element_by_link_text(): This finds a single element by a link identified by the link text. Multiple WebElements can be located using find_elements_by_link_text(). 
• find_element_by_partial_link_text(): This finds a single element by a link identified by the partial text the element is carrying. Multiple WebElements can be located using find_elements_by_partial_link_text().
• find_element_by_xpath(): This finds a single element by providing an XPath expression. Multiple WebElements can be located using find_elements_by_xpath(). 
• find_element_by_css_selector(): This finds a single element by providing CSS selectors. Multiple WebElements can be located using find_elements_by_css_selector(). 

Now, let's find the input box using find_element_by_id():

searchBox = driver.find_element_by_id('search_query_top')
#searchBox = driver.find_element_by_xpath('//*[@id="search_query_top"]')
#searchBox = driver.find_element_by_css_selector('#search_query_top')

As you can see in the preceding code, searchBox can be located using any convenient locators that are provided with their respective arguments.

The WebElement that is obtained can be accessed for the following properties and general methods, as well as many more:

• get_attribute(): This returns the attribute value for the key argument provided, such as value, id, name, and class.
• tag_name: This returns the HTML tag name of a particular WebElement.
• text: This returns the text of the WebElement.
• clear(): This clears the text of HTML form elements.
• send_keys(): This is used to fill with text and provide the key effect, such as pressing ENTER, BACKSPACE, and  DELETE, available from the selenium.webdriver.common.keys module in selenium.webdriver.common to the HTML form elements.
• click(): This performs the clicking action to the WebElement. This is used for HTML elements such as Submit Button.

In the following code, we will be using the functions and properties listed previously in searchBox:

print("Type :",type(searchBox))
<class 'selenium.webdriver.remote.webelement.WebElement'>

print("Attribute Value :",searchBox.get_attribute("value")) #is empty
Attribute Value : 

print("Attribute Class :",searchBox.get_attribute("class"))
Attribute Class : search_query form-control ac_input

print("Tag Name :",searchBox.tag_name)
Tag Name : input

Let's clear the text inside searchBox and input the text Dress to be searched. We also need to submit the button located on the right-hand side of the searchBox and click it to execute the search using the WebElement method, click():

searchBox.clear() 
searchBox.send_keys("Dress")
submitButton = driver.find_element_by_name("submit_search")
submitButton.click()

The browser will process the search action for the submitted text Dress and load the results page.

Now that the search action is complete, to verify the successful search, we will extract information regarding the product numbers and count using the following code:

#find text or provided class name
resultsShowing = driver.find_element_by_class_name("product-count")
print("Results Showing: ",resultsShowing.text) 

Results Showing: Showing 1-7 of 7 items

#find results text using XPath
resultsFound = driver.find_element_by_xpath('//*[@id="center_column"]//span[@class="heading-counter"]')
print("Results Found: ",resultsFound.text)

Results Found: 7 results have been found.

With the number of items and the count of the products that were found, this conveys a successful message to our search process. Now, we can proceed with looking for products using XPath, CSS selectors, and so on:

#Using XPath
products = driver.find_elements_by_xpath('//*[@id="center_column"]//a[@class="product-name"]')

#Using CSS Selector
#products = driver.find_elements_by_css_selector('ul.product_list li.ajax_block_product a.product-name')

foundProducts=[]
for product in products:
    foundProducts.append([product.text,product.get_attribute("href")])

From the preceding code,  products obtained is iterated and an individual item is added to the Python list foundProducts. product is an object of WebElement, in other words, selenium.webdriver.remote.webelement.WebElement,  while properties are collected using text and get_attribute():

print(foundProducts) 

[['Printed Summer Dress',
'http://automationpractice.com/index.php?id_product=5&controller=product&search_query=Dress&results=7'],
['Printed Dress',
'http://automationpractice.com/index.php?id_product=4&controller=product&search_query=Dress&results=7'],
['Printed Summer Dress',
'http://automationpractice.com/index.php?id_product=6&controller=product&search_query=Dress&results=7'],
['Printed Chiffon Dress',
'http://automationpractice.com/index.php?id_product=7&controller=product&search_query=Dress&results=7'],['PrintedDress',
'http://automationpractice.com/index.php?id_product=3&controller=product&search_query=Dress&results=7'],
['Faded Short Sleeve T-shirts',
'http://automationpractice.com/index.php?id_product=1&controller=product&search_query=Dress&results=7'],['Blouse',
'http://automationpractice.com/index.php?id_product=2&controller=product&search_query=Dress&results=7']]

In this section, we explored the various properties and methods from selenium.webdriver that are used to deal with the browser, use HTML forms, read page content, and so on. Please visit https://selenium-python.readthedocs.io for more detailed information on Python Selenium and its modules. In the next section, we will use most of the methodologies that were used in the current section to scrape information from a web page.