Common JavaScript Libraries

Although the core JavaScript language is important to know, you can’t get very far on the modern Web without using at least one of the language’s many third-party libraries. You might see one or more of these commonly used libraries when looking at page source code.

Executing JavaScript using Python can be extremely time consuming and processor intensive, especially if you’re doing it on a large scale. Knowing your way around JavaScript and being able to parse it directly (without needing to execute it to acquire the information) can be extremely useful and save you a lot of headaches. 

<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></
 script>

<!-- Google Analytics -->
<script type="text/javascript">

var _gaq = _gaq || []; 
_gaq.push(['_setAccount', 'UA-4591498-1']);
_gaq.push(['_setDomainName', 'oreilly.com']);
_gaq.push(['_addIgnoredRef', 'oreilly.com']);
_gaq.push(['_setSiteSpeedSampleRate', 50]);
_gaq.push(['_trackPageview']);

(function() { var ga = document.createElement('script'); ga.type =
'text/javascript'; ga.async = true; ga.src = ('https:' ==
document.location.protocol ? 'https://ssl' : 'http://www') +
'.google-analytics.com/ga.js'; var s =
document.getElementsByTagName('script')[0];
s.parentNode.insertBefore(ga, s); })();

</script>

var marker = new google.maps.Marker({
      position: new google.maps.LatLng(-25.363882,131.044922),
      map: map,
      title: 'Some marker text'
  });

Executing JavaScript in Python with Selenium

Selenium is a powerful web scraping tool developed originally for website testing. These days it’s also used when the accurate portrayal of websites—as they appear in a browser—is required. Selenium works by automating browsers to load the website, retrieve the required data, and even take screenshots or assert that certain actions happen on the website. 

Selenium does not contain its own web browser; it requires integration with third-party browsers in order to run. If you were to run Selenium with Firefox, for example, you would literally see a Firefox instance open up on your screen, navigate to the website, and perform the actions you had specified in the code. Although this might be neat to watch, I prefer my scripts to run quietly in the background, so I use a tool called PhantomJS in lieu of an actual browser. 

PhantomJS is what is known as a “headless” browser. It loads websites into memory and executes JavaScript on the page, but does it without any graphic rendering of the website to the user. By combining Selenium with PhantomJS, you can run an extremely powerful web scraper that handles cookies, JavaScript, headers, and everything else you need with ease.

You can download the Selenium library from its website or use a third-party installer such as pip to install it from the command line.

PhantomJS can be downloaded from its website. Because PhantomJS is a full (albeit headless) browser and not a Python library, it does require a download and installation to use and cannot be installed with pip.

Although there are plenty of pages that use Ajax to load data (notably Google), I’ve created a sample page at http://bit.ly/1HYuH0L to run our scrapers against. This page contains some sample text, hardcoded into the page’s HTML, that is replaced by Ajax-generated content after a two-second delay. If we were to scrape this page’s data using traditional methods, we’d only get the loading page, without actually getting the data that we want.

The Selenium library is an API called on the a WebDriver. The WebDriver is a bit like a browser in that it can load websites, but it can also be used like a BeautifulSoup object to find page elements, interact with elements on the page (send text, click, etc.), and do other actions to drive the web scrapers. 

The following code retrieves text behind an Ajax “wall” on the test page:

from selenium import webdriver
import time

driver = webdriver.PhantomJS(executable_path='')
driver.get("http://pythonscraping.com/pages/javascript/ajaxDemo.html")
time.sleep(3)
print(driver.find_element_by_id("content").text)
driver.close()

This creates a new Selenium WebDriver, using the PhantomJS library, which tells the WebDriver to load a page and then pauses execution for three seconds before looking at the page to retrieve the (hopefully loaded) content. 

Depending on the location of your PhantomJS installation, you might also need to explicitly point Selenium in the right direction when creating a new PhantomJS WebDriver:

driver = webdriver.PhantomJS(executable_path='/path/to/download/
                             phantomjs-1.9.8-macosx/bin/phantomjs')

If everything is configured correctly the script should take a few seconds to run and result in the following text:

Here is some important text you want to retrieve!
A button to click!

Note that although the page itself contains an HTML button, Selenium’s .text function retrieves the text value of the button in the same way that it retrieves all other content on the page. 

If the time.sleep pause is changed to one second instead of three, the text returned changes to the original:

This is some content that will appear on the page while it's loading.
 You don't care about scraping this.

Although this solution works, it is somewhat inefficient and implementing it could cause problems on a large scale. Page load times are inconsistent, depending on the server load at any particular millisecond, and natural variations occur in connection speed. Although this page load should take just over two seconds, we’re giving it an entire three seconds to make sure that it loads completely. A more efficient solution would repeatedly check for the existence of some element on a fully loaded page and return only when that element exists. 

This code uses the presence of the button with id loadedButton to declare that the page has been fully loaded: from selenium import webdriver.

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.PhantomJS(executable_path='')
driver.get("http://pythonscraping.com/pages/javascript/ajaxDemo.html")
try:
    element = WebDriverWait(driver, 10).until(
                       EC.presence_of_element_located((By.ID, "loadedButton")))
finally:
    print(driver.find_element_by_id("content").text)
    driver.close()

There are several new imports in this script, most notably WebDriverWait and expected_conditions, both of which are combined here to form what Selenium calls an implicit wait.

An implicit wait differs from an explicit wait in that it waits for some state in the DOM to occur before continuing, while an explicit wait defines a hardcoded time like in the previous example, which has a wait of three seconds. In an implicit wait, the triggering DOM state is defined by expected_condition (note that the import is cast to EC here, a common convention used for brevity). Expected conditions can be many things in the Selenium library, among them: 

• An alert box pops up
• An element (such as a text box) is put into a “selected” state
• The page’s title changes, or some text is now displayed on the page or in a specific element
• An element is now visible to the DOM, or an element disappears from the DOM

Of course, most of these expected conditions require that you specify an element to watch for in the first place. Elements are specified using locators. Note that locators are not the same as selectors (see previous sidebar for more on selectors). A locator is an abstract query language, using the By object, which can be used in a variety of ways, including to make selectors. 

In the following example code, a locator is used to find elements with the id loadedButton:

EC.presence_of_element_located((By.ID, "loadedButton"))

Locators can also be used to create selectors, using the find_element WebDriver function:

print(driver.find_element(By.ID, "content").text)

Which is, of course, functionally equivalent to the line in the example code:

print(driver.find_element_by_id("content").text)

If you do not need to use a locator, don’t; it will save you an import. However, it is a very handy tool that is used for a variety of applications and has a great degree of flexibility. 

The following locator selection strategies can used with the By object:

ID

Used in the example; finds elements by their HTML id attribute.

CLASS_NAME

Used to find elements by their HTML class attribute. Why is this function CLASS_NAME and not simply CLASS? Using the form object.CLASS would create problems for Selenium’s Java library, where .class is a reserved method. In order to keep the Selenium syntax consistent between different languages, CLASS_NAME was used instead.

CSS_SELECTOR

Find elements by their class, id, or tag name, using the #idName, .className, tagName convention.

LINK_TEXT

Finds HTML <a> tags by the text they contain. For example, a link that says “Next” can be selected using (By.LINK_TEXT, "Next").

PARTIAL_LINK_TEXT

Similar to LINK_TEXT, but matches on a partial string.

NAME

Finds HTML tags by their name attribute. This is handy for HTML forms.

TAG_NAME

Finds HTML tags by their tag name.

XPATH

Uses an XPath expression (the syntax of which is described in the upcoming sidebar) to select matching elements.