Let's now build a simple Weather Reporter application. The weather data for any given location will be fetched from the network, suitably formatted, and presented to the user.

We will use a higher level module named urllib to fetch weather data from the web. The urllib module is part of Python's standard library and it provides an easy to use API for working with URLs. It has four submodules:

• urllib.request: For opening and reading URLs
• urllib.error: For handling exceptions raised by urllib.request
• urllib.parse: For parsing URLs
• urllib.robotparser: For parsing robots.txt files

With urllib.request, fetching the contents of a web page turns into three lines of code   (see 9.05_urllib_demo.py):

import urllib.request

with urllib.request.urlopen('http://www.packtpub.com/') as f:
  print(f.read())

This prints the entire HTML source code or whatever is the response from the web page http://www.packtpub.com. This is, in essence, the core of mining the web for information.

Now that we know how to get data from a URL, let's apply it to building our Weather Reporter application. This application should take the location as an input from the user and fetch relevant weather-related data, as shown in the following screenshot:

We create a class, WeatherReporter, and call it from outside the class within the mainloop (see the code of 9.06_weather_reporter.py):

def main():
  root=Tk()
  WeatherReporter(root)
  root.mainloop()

if __name__ == '__main__':
  main()

We do not discuss the code for creating this GUI here, as we have done similar coding many times in all the the previous chapters. The weather data is displayed on a canvas (see 9.06_weather_reporter.py):

When you specify a location and click on the Go button, it calls a command callback named on_show_weather_button_clicked.

We then fetch the weather data from a website.

There are two ways to fetch data from a website. The first method involves getting an HTML response from a website, and then parsing the received HTML response for data that is relevant to us. This type of data extraction is called site scraping.

Scrapy and Beautiful Soup are two popular site-scraping frameworks for extracting data from websites. You can find the official documentation of the two libraries at http://scrapy.org/ and http://www.crummy.com/software/BeautifulSoup/.

Site scraping is a rather crude method that is employed only when a given website does not provide a structured way to retrieve data. On the other hand, some websites are willing to share data through a set of APIs, provided you query it for data using the specified URL structure. This is clearly more elegant than site scraping, because data is interchanged in a reliable and mutually agreed format.

For our Weather Reporter application, we want to query some weather channels for a given location, and in turn retrieve and display the data on our canvas.

Fortunately, there are several weather APIs that we can use. In our example, we will use the weather data provided by the following website:
http://openweathermap.org/

In order to use the API, you need to sign up for a free API key here:
http://home.openweathermap.org/users/sign_up

The OpenWeatherMap service provides free weather data and forecast APIs. This site collates weather data from more than 40,000 weather stations across the globe, and the data can be assessed by city name and geographic coordinates, or their internal city ID.

The website provides weather data in two data formats:

• JSON (JavaScript Object Notation)
• XML (Extensible Markup Language)

XML and JSON are two popular interchangeable data serialization formats widely used for interchanging data between different applications, which may be running on different platforms and using different programming languages, thus providing the benefit of interoperability.

JSON is simpler than XML, because its grammar is simpler and it maps more directly onto the data structures used in modern programming languages. JSON is better suited for exchanging data, but XML is good for exchanging documents.

The API documentation tells us of a query such as:

api.openweathermap.org/data/2.5/weather?q=London,uk&APPID={APIKEY} 

The preceding code returns weather data for London in JSON format as follows:

{"coord":{"lon":-0.12574,"lat":51.50853},"sys":{"country":"GB","sunrise":1377147503,"sunset":1377198481},"weather":[{"id":500,"main":"Rain", "description": "light rain","icon":"10d"}],"base":"gdps stations","main":{"temp":294.2, "pressure":1020,"humidity":88, "temp_min":292.04,"temp_max":296.48},"wind":{"speed":1,"deg":0},"rain":{"1h":0.25},"clouds":{"
 all":40},"dt":1377178327,"id":2643743,"name":"London","cod":200}

The syntax of JSON is simple. Any piece of JSON data is a name/value pair and each piece of data is separated from the others by commas. JSON uses curly braces {} to hold objects and square brackets [ ] to hold arrays. Accordingly, we define a method to get the weather data in JSON format in our application (see 9.06_weather_reporter.py):

def get_data_from_url(self):
 try:
   params = urllib.parse.urlencode( {'q': self.location.get(), 'APPID': self.APIKEY},
    encoding="utf-8")
   api_url = ('http://api.openweathermap.org/data/2.5/weather?{}'.format(params))
   with urllib.request.urlopen(api_url) as f:
     json_data = f.read()
     return json_data
 except IOError as e:
   messagebox.showerror('Unable to connect', 'Unable to connect %s' % e)
   sys.exit(1)

The description for the code is as follows:

• This method uses urllib to retrieve responses from the website. It returns the response in JSON format.
• Now, we'll start processing the JSON data. The weather data returned using the API is encoded in JSON format. We need to convert this data into the Python data type. Python provides a built-in json module that simplifies the process of encoding/decoding JSON data. We therefore import the json module into our current namespace.

Then, we'll use this module to convert the retrieved JSON data into the Python dictionary format (see 9.06_weather_reporter.py):

def json_to_dict(self, json_data):
 decoder = json.JSONDecoder()
 decoded_json_data = decoder.decode(json_data.decode("utf-8"))
 flattened_dict = {}
 for key, value in decoded_json_data.items():
   if key == 'weather':
     for ke, va in value[0].items():
       flattened_dict[str(ke)] =  str(va).upper()
     continue
   try:
     for k, v in value.items():
     flattened_dict[str(k)] = str(v).upper()
   except:
     flattened_dict[str(key)] = str(value).upper()
 return flattened_dict 

Now that we have a dictionary of all weather-related information provided by the API, we simply display the retrieved weather data using canvas.create_text and canvas.create_image. The code for displaying the weather data is self-explanatory (see 9.06_weather_reporter.py).

Our Weather Reporter application is now functional.

To summarize the code for the Weather Reporter, we use the urllib module to query the weather API provided by our data provider. The data is fetched in JSON format. The JSON data is then decoded into a Python-readable format (dictionary).

The converted data is then displayed on the canvas using the create_text and create_image methods.