Searching Google for getting information about something seems to be an everyday activity for many people. Let's try to search Google for some information.
This recipe uses a third-party Python library, requests, which can be installed via pip, as shown in the following command:
$ pip install SOAPpy
Google has sophisticated APIs to conduct a search. However, they require you to register and get the API keys by following a specific way. For simplicity's sake, let us use Google's old plain Asynchronous JavaScript (AJAX) API to search for some information about Python books.
Listing 8.6 gives the code for searching Google for custom information, as shown:
#!/usr/bin/env python
# Python Network Programming Cookbook -- Chapter - 8
# This program is optimized for Python 2.7.# It may run on any other version with/without modifications.
import argparse
import json
import urllib
import requests
BASE_URL = 'http://ajax.googleapis.com/ajax/services/search/web?v=1.0'
def get_search_url(query):
return "%s&%s" %(BASE_URL, query)
def search_info(tag):
query = urllib.urlencode({'q': tag})
url = get_search_url(query)
response = requests.get(url)
results = response.json()
data = results['responseData']
print 'Found total results: %s' %
data['cursor']['estimatedResultCount']
hits = data['results']
print 'Found top %d hits:' % len(hits)
for h in hits:
print ' ', h['url']
print 'More results available from %s' %
data['cursor']['moreResultsUrl']
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Search info from
Google')
parser.add_argument('--tag', action="store", dest="tag",
default='Python books')
# parse arguments
given_args = parser.parse_args()
search_info(given_args.tag)If you run this script by specifying a search query in the --tag argument, then it will search Google and print a total results count and the top four hits pages, as shown:
$ python 8_6_search_products_from_Google.py Found total results: 12300000 Found top 4 hits: https://wiki.python.org/moin/PythonBooks http://www.amazon.com/Python-Languages-Tools-Programming- Books/b%3Fie%3DUTF8%26node%3D285856 http://pythonbooks.revolunet.com/ http://readwrite.com/2011/03/25/python-is-an-increasingly-popu More results available from http://www.google.com/search?oe=utf8&ie=utf8&source=uds&start=0&hl=en &q=Python+books
In this recipe, we defined a short function, get_search_url(), which constructs the search URL from a BASE_URL constant and the target query.
The main search function, search_info(), takes the search tag and constructs the query. The requests library is used to make the get() call. The returned response is then turned into JSON data.
The search results are extracted from the JSON data by accessing the value of the 'responseData' key. The estimated results and hits are then extracted by accessing the relevant keys of the result data. The first four hit URLs are then printed on the screen.