Chapter 2: Using Remote Data in Your Mapplets
You now have Mapplet basics down. In the previous chapter, you learned how to plot markers on a map using static content, or content that never changes. It is now time to make your application more interactive by using dynamic data, or data that is retrieved from remote servers.
Mapping data sets that don't often change (if ever), such as a chain store's locations or a series of historical buildings, may not require dynamic retrieval. However, when you start to map information that changes often, such as news or event data (like concerts), you'll need to periodically retrieve the data to refresh your application. In this chapter, you'll learn how to do just that.
Specifically, you will explore the following:
- Typical data formats used in data feeds and web services
- Google's JavaScript APIs to fetch remote content
- Techniques for making Mapplets more interactive and responsive
- Tools for plotting dynamic content with Mapplets
Data Feed Formats
Before you get started retrieving remote content, I'll discuss the data formats. You'll see plenty of acronyms for data formats: CSV, XML, RSS, Atom, KML, JSON.... These are a few of the popular flavors of the moment, and you might retrieve any of them from a remote server or with a web service. The following sections highlight the ones you'll be using in this book.
CSV: The Equivalent of Cave Paintings
Comma-separated values (CSV) is an ancient file format that you've most likely encountered as an export type for spreadsheets and flat-file databases. It is a plain-text data format that uses commas to delimit simple fields in a text file.
It's worth a mention, though it's not as commonly used on the Web as some of the other formats mentioned next. You can imagine the fields in a data file that you'd need to store some mapping/marker information: the marker description (in this case, the band and venue name), the latitude, and the longitude. Data in a CSV file looks like this:
the redwalls, Knitting Factory, 40.7173, −74.0053, 
dirty projectors, Bowery Ballroom, 40.7205, −73.9937XML: The Worldwide Format
Many publishing and data feed formats are based on Extensible Markup Language (XML), including RSS, Atom, KML, and so on. Most data feeds on the Internet today are some flavor of XML. All modern browsers have built-in support for parsing XML data using JavaScript APIs.
Note You can find more information about XML at the W3C: http://www.w3.org/XML/.
You'll be using XML with most of the examples in this book, so let's look at a simple XML file that stores the sample marker data, as in the previous example. This is some simple XML that you can use to store concert data:
<?xml version="1.0" encoding="UTF-8"?>
<markers>
<marker title="the redwalls" venue="Knitting Factory"
lat="40.7173" lng="−74.0053"/>
<marker title="dirty projectors" venue="Bowery Ballroom"
lat="40.7205" lng="−73.9937"/>
</markers>RSS: A Timely Format
RDF Site Summary or Really Simple Syndication (RSS)—take your pick—is used throughout the Web to syndicate web content such as blogs and news feeds. RSS, which is a flavor of XML, is typically referred to as a feed or data feed.
RSS typically stores summaries of blog or news content with links to the content, but RSS is also widely used to syndicate other types of content ranging from event data such as Tourfilter and Upcoming.org to social status updates such as Twitter and Facebook's Status Updates.
Note You can find more information about RSS and the RSS 2.0 specification at http://cyber.law.harvard.edu/rss/rss.html.
For an example of RSS used to syndicate event data, let's take a look at Tourfilter's RSS feed of concert data in New York, shown here. Notice the header information such as title, link, and description. Then the RSS contains a "feed" of "items" that may contain a title, a publication date (<pubDate>), and a link. This RSS feed doesn't contain any specific location information such as latitude and longitude, which would make it a GeoRSS feed. (For more information about GeoRSS, check out http://www.georss.org.)
<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
<channel>
<title>tourfilter new york shows</title>
<link>http://www.tourfilter.com/newyork</link>
<description>Get an email when bands you like come to New York:
A community calendar of upcoming local live music.</description>
<language>en</language>
<item>
<title>hot rod circuit (Knitting Factory 12/9)</title>
<pubDate>Fri, 12 Oct 2007 04:49:09 −0400</pubDate>
<link>http://www.tourfilter.com/newyork/hot_rod_circuit</link>
</item>
<item>
<title>Michelle Shocked (Highline Ballroom 12/9)</title>
<pubDate>Mon, 01 Oct 2007 04:50:43 −0400</pubDate>
<link>http://www.tourfilter.com/newyork/michelle_shocked</link>
</item>
Ib
</channel>
</rss>JSON: The Programmer-Friendly Format
So, what is this JavaScript Object Notation (JSON) that we all keep hearing about? JSON is your friend, so it's time to learn it and love it. JSON is not necessarily a data feed format but is really a data interchange format. It is basically a simple text format that is a chunk of valid JavaScript that lets you represent data as JavaScript objects. JSON is easy to read and write, which makes it easy to program and debug (by you). And, it's a subset of the JavaScript language, so it's easy to parse data represented as JSON within your JavaScript code.
You can use JSON to store arrays, as well as key/values pairs (typically referred to as a hash table, dictionary, or associative array), which are universal data structures. So, this is a format that can be used across many programming languages and environments.
JSON is becoming widely adopted, especially with the rise of Ajax applications that consume web services, because it is easy to use within these JavaScript-based applications. More and more companies that offer web services, such as Google and Yahoo, are offering JSON output as an alternative to XML.
Let's now see what JSON looks like and how you can use it within JavaScript. Using some of the data from the previous CSV and XML examples, here is the same map marker data in JSON:
{
"markers": [
{
"title": "the redwalls",
"venue": "Knitting Factory",
"lat": 40.7173,
"lng": −74.0053
},
{
"title": "dirty projectors",
"venue": "Bowery Ballroom",
"lat": 40.7205,
"lng": −73.9937
}
]
}In this example, you are storing the array of Markers, denoted by brackets: [ ]. Each marker has a few attributes (title, venue, latitude, and longitude) that are stored as key/values pairs (for example, "title": 'the redwalls"). Pretty readable, right?
Tip json.org is great resource for all things JSON. The site provides a good introduction of JSON as well as links to many JSON libraries for a variety of languages. You can also find some examples of JSON formatting and data structures at http://json.org/example.html.
Retrieving Remote Content with Mapplet APIs
Google provides three APIs for retrieving remote content for Mapplets. These APIs are part of the Gadget API suite and can be used in both Mapplets and Gadgets. Using the APIs, you can retrieve remote data as text, XML, or JSON. The three API calls are as follows:
_IG_FetchContent()takes a URL as an argument and returns the text from the given URL. This API should be used when working with text or HTML content._IG_FetchXmlContent()takes a URL as an argument and returns the XML content as a DOM object. This should be used when retrieving XML content._IG_FetchFeedAsJSON()should be used when retrieving RSS or Atom feeds. This takes a URL as an argument and returns the feed as a JSON object.
Note I won't be covering the _IG_FetchFeedAsJSON() API in this chapter. It is a useful API when dealing with RSS and Atom feeds, but this API currently doesn't allow you to parse geographic fields, such as latitude and longitude, from RSS or Atom feeds.
You can find more information about these APIs and fetching remote content within Mapplets and Gadgets at http://code.google.com/apis/gadgets/docs/remote-content.html.
Retrieving Text
Let's return to some code examples! You'll start by using the first of Google's "fetch" APIs to retrieve some text from a remote server and display it in the Mapplet. Text in this case can refer to any type of text: HTML, CSV data, or even plain, unformatted text. The API_IG_FetchContent() will retrieve the remote URL you specify and return the data from the URL as plain text. It's up to your application to use the retrieved text data as needed. If you retrieve HTML, you may want to display the chunk of HTML you've retrieved in your application's user interface. If you retrieve something like CSV-formatted data, you will probably be parsing the data and using some of this data in various parts of your application, such as creating markers on a map.
In the first example, you will fetch some HTML content and display the HTML in a JavaScript alert() window. It's a simple example but will show you how the first of Google's "fetch" APIs work.
- In the Mapplet Scratch Pad at http://maps.google.com/maps/mm, click the "Go back to editor" button to display the scratch pad editor.
- Cut and paste the Mapplet XML from
example_2_1_fetch_text.xml, and paste it into the Mapplet Scratch Pad. - Click the Preview button in the scratch pad. You should see the Mapplet shown in Figure 2-1.
Figure 2-1. HTML from Apress.com displayed in an alert window
Examining the Code
Congratulations—your Mapplet has successfully fetched some remote content! In this example, you used the _IG_FetchContent() API to retrieve the HTML from www.apress.com. _IG_FetchContent() takes two arguments: the URL that you want to fetch and a function that gets called once the content from the URL has been fetched. Remember, this is referred to as a callback. Let's look at this example line by line.
You first create a variable called url that contains the Apress home page URL: var url = "http://www.apress.com";. To fetch the Apress home page, you use the following code:
_IG_FetchContent(url, handleFetchContent);_IG_FetchContent() is an asynchronous function, meaning it will return immediately after it is called and will call the handleFetchContent function (the callback) after the remote content has been fetched.
Tip It is important to remember that the "fetch" APIs are asynchronous and may take some time to return the remote content that you are fetching. Make sure your application takes this into consideration. Since the content you are fetching may not be returned immediately, you may want to give some notification to your user that you are retrieving content. I'll show you a few ways of doing of this in the next few examples.
Once the remote content has been fetched, _IG_FetchContent() will call the function handleFetchContent with a JavaScript string object containing the text you've just fetched. In handleFetchContent, you are using the variable responseText to store the fetched text. If an error occurs in retrieving the remote data, the returned text will be null. Notice that you are checking for this and displaying a simple error if it happens. Finally, you display the first 200 characters of the text using the following code:
alert(responseText.substr(0,200));.
function handleFetchContent (responseText) {
// check to see if there was an error in the fetch
if (responseText == null) {
// an error occurred
alert("failed to retrieve data from " + url);
return;
}
// display the text that was returned from apress.com
// but only display the first 200 characters
alert(responseText.substr(0,200));
}In the end, you've fetched and displayed the first 200 HTML characters of Apress.com.
Making Mapplets More Interactive
Let's expand on the previous example by retrieving some map-related data; at the same time, you'll make the application more interactive. The previous example didn't give the user any indication you were loading data in the background. You may not have noticed any delay in retrieving and displaying the data, but it's always important to take this possibility into consideration and give your user some indication that the initial data is loading.
With this next example, I'll introduce a few new concepts. First, let's create a link in the application that allows the user to load the remote data with a single click of a link. Second, you will create a status indication in the application to let users know what's going on behind the scenes: whether the data is loading, whether an error has occurred, or whether the data was loaded successfully. Last, let's take the remote data you are fetching and actually display it in your application. You aren't going to add anything to the map itself quite yet—that's coming in one of the next examples.
You haven't focused much on the application portion of the Mapplet yet—the small web page that is displayed in the bottom-left portion of the screen when you are viewing the Mapplet. All you have done so far is use <h2> tags to display the title of the sample applications. Let's expand on this in the following example, adding a simple link to load the remote data:
<a href="#" onclick="loadData();">Load Data</a>You will also add two HTML tags to the application to display status messages as well as content that you retrieve from the server. These will look like this:
<p>status: <span id="status">Nothing loaded yet.</span></p>
<p id="content"></p>Notice the IDs that you've assigned to both the status tag and the content tag. You'll use these IDs to dynamically update the HTML contained within both the status and content tags. I'll explain this after you take a look at the code.
You are also adding some simple CSS to help style the web page portion of the Mapplet. You'll see how easy it is to style your application; in the following example, you are modifying only the font.
In this next example, you will be retrieving data from your own server. As you move forward in this book, a lot of the sample code will need to run from a web server that can be accessible on the Net—not your local machine. In this next example, you'll create a simple CSV data file to store some band and venue information, and the Mapplet will fetch this data from the server. If you don't have access to your own server, you can upload the test files to Google Pages (http://pages.google.com) for the next few examples, but you'll need to get access to a server with PHP for some of the upcoming examples in the next chapters.
- Place the file markers.csv on a public web server that is publicly available. If you don't have access to a public web server, you can upload the markers.csv file to your account at http://pages.google.com. The data in markers.csv looks like this:
the redwalls, Knitting Factory, 40.7173, −74.0053,
dirty projectors, Bowery Ballroom, 40.7205, −73.9937 - In the Mapplet Scratch Pad, click the "Go back to editor" button to display the scratch pad editor.
- Edit the Mapplet source in
example_2_2_fetch_csv.xml, and make sure the URL in the following line points to yourmarkers.csvfile:var url = "http://yourserver.com/path/to/markers.csv"; - After editing
example_2_2_fetch_csv.xml, copy and paste the contents of the file into the Mapplet Scratch Pad. - Click the Preview button in the scratch pad. You should see the Mapplet shown in Figure 2-2.
- Once the Mapplet loads, click the Load Data link to load the remote marker data.
In this example, you again used the _IG_FetchContent() API to retrieve some content from your own server and then displayed band and venue information in your Mapplet. I'll now walk you through this example, covering the new additions.
You added some simple CSS to the application, changing the font to Arial 12 pixels by placing some CSS in the Mapplet code:
<style type="text/css">
<!--
body,div,span,p {
font-family: arial, sans-serif;
font-size: 12px;
}
-->
</style>To help make the application more interactive, you added a simple link that loads the data when clicked. Using the onclick attribute of the anchor tag, the loadData() function is called whenever a user clicks the Load Data link:
<p><a href="#" onclick="loadData();">Load Data</a><p>Next, you added two new sections to help you display the application's status as well as the content that you've fetched:
<p>status: <span id="status">Nothing loaded yet. Click Load Data
above.</span></p>
<p id="content"></p>To update the status and content, you're using the function _gel() to get access to the DOM element that you need to update (status and content in this case). _gel() is Google's wrapper for the JavaScript function getElementById(), which gives you the DOM element for the specified ID. So, to update the HTML between the <span id="status"> and </span> tags, you use the following:
_gel("status").innerHTML = "loading...";Notice that you update the status with "loading" when you first call loadData() and then update it again if either an error occurs when fetching the data or you successfully fetch the remote data.
After you successfully fetch the band and venue data, you parse the data (which is one long string of comma-delimited fields) into an array using the JavaScript split() function. Once you have the data in the array markerData, you loop over the array, adding each field to the displayHTML variable. Notice that you need to add a line break (<br/>) after every fourth field is pulled from the array; you do this since the marker data holds four fields for every marker: band name, venue name, latitude, and longitude.
// use the split to parse the incoming marker data
var markerData = responseText.split(",");
// loop over the individual marker data fields
for (var i = 0; i < markerData.length; i++) {
//...
}You finally update the application's content section with the following command:
_gel("content").innerHTML = displayHTML;Retrieving XML
Chances are good that most of the third-party data that you'll be dealing with when creating mashups will be some flavor of XML. Fortunately, using Google's _IG_FetchXmlContent() API as well as built-in JavaScript functions to parse the XML, it's fairly easy to use XML content in Mapplets. Over the next few examples, you'll see how to create a Mapplet that fetches remote XML data.
This next example is similar to the previous example. You will be retrieving the same data as XML instead of CSV. You'll see how to use the Google API to fetch XML content as well as learn how to parse XML within your Mapplet. This example won't touch the map, but you will get there in the next example—I promise! I know you've been waiting.
- Place the file markers.xml on a public web server. Again, if you don't have access to a public web server, you can upload the file to your account at http://pages.google.com. The data in markers.xml looks like the following:
<?xml version="1.0" encoding="UTF-8"?>
<markers>
<marker title="the redwalls" venue="Knitting Factory"
lat="40.7173" lng="−74.0053" />
<marker title="dirty projectors" venue="Bowery Ballroom"
lat="40.7205" lng="−73.9937" />
</markers> - In the Mapplet Scratch Pad, click the "Go back to editor" button to display the scratch pad editor.
- Edit the Mapplet source in
example_2_3_fetch_xml.xml, and make sure the URL in the following line points to yourmarkers.xmlfile:var url = "http://yourserver.com/path/to/markers.xml"; - After editing
example_2_3_fetch_xml.xml, copy and paste the contents of the file into the Mapplet Scratch Pad. - Click the Preview button in the scratch pad. You should see the Mapplet shown in Figure 2-3.
- Once the Mapplet loads, click the Load Data link to load the remote marker data.
So, there you have it—you just retrieved and parsed some XML from a remote server! Let's take a quick look at what's new in this one.
First, you should have changed the url to point to your own markers.xml file:
var url = "http://yourserver.com/path/to/markers.xml";Next, inside the loadData() function, you called the Google API to fetch XML content: _IG_FetchXmlContent(). Like the API to fetch text, this takes a URL to an XML document as an argument and returns the XML as a DOM object. You can then use standard JavaScript functions to extract the data from the DOM object.
_IG_FetchXmlContent(url, handleFetchContent);Once the XML content is fetched, the Google API will call the function handleFetchContent(), passing the fetched data into the function. At the top of handleFetchContent(), you do some error checking to make sure the data is valid XML. The first check is to make sure response is not null. Second, you check whether the response is a valid JavaScript object. Last, you use the DOM attribute firstChild to ensure response is a valid DOM object. If any of those checks fail, something went wrong in fetching the data or the data isn't valid XML.
if (response == null || typeof(response) != "object" ||
response.firstChild == null) {
_gel("status").innerHTML = "Failed to load valid XML data";
return;
}If you have valid XML to work with, the previous validation checks will all pass with flying colors. Next, you use the getElementsByTagName() DOM function to get each of the markers from the XML. getElementsByTagName() will search the DOM object and return an array of DOM elements matching the name you are looking for (marker in this case). You use the following line to get each of the marker elements:
var markers = response.getElementsByTagName("marker");The variable markers now contains an array of DOM elements. Next, it's a matter of looping over the markers array, extracting all the information for each marker—band/title, venue name, latitude, and longitude. Since all this information was stored as attributes in the marker XML elements (<marker title="the redwalls" venue="Knitting Factory" lat="40.7173" lng="−74.0053"/>), you can use the DOM function getAttribute() to extract each of the fields from the marker element. After you extract each of the fields, you add the title and venue to the displayHTML variable, which is then rendered to the screen of the application.
for (var i = 0; i < markers.length; i++) {
var marker = markers[i];
var title = marker.getAttribute("title");
var venue = marker.getAttribute("venue");
var lat = marker.getAttribute("lat");
var lng = marker.getAttribute("lng");
displayHTML += "<div>" + title + " - " + venue + "</div>";
}Notice that you didn't do anything with the latitude and longitude? That's coming in the next example, when you combine everything you've learned from this chapter with some of the Mapplet APIs from the previous chapter. Are you ready? Good, let's move on.
Mapping Remote XML Data
At this point, you have a lot of tools that you can pull together into a single application. You now know how to retrieve XML content from a remote server, as well as plot data on a map using the Mapplet APIs. Let's use all your new skills to plot some remote data on a map! It's about time, right? You'll even add a little interactivity so that your map responds to clicks from your user.
You'll use some of the code from the previous example, retrieving the same marker XML data from your own server. But, instead of just displaying the list of band and venues in the left panel of your Mapplet, let's actually map the venues on the map. You'll also use code from the previous chapter to help you create the markers on the map and center the map so that you can view all the markers on the map at the same time.
Before you get started with the code, let's first take a look at the Mapplet you are going to create. Notice in Figure 2-4 that you have a list of the bands and venues on the left side and markers placed on the maps for the venue locations? You're adding some new interactivity to this application. When you click one of the band/venue links on the left, the info window for that corresponding marker will open on the map. It's always a good idea to let your user navigate the content through either the list or the map.
Figure 2-4. XML data displayed on the map
Follow these steps to view the latest Mapplet:
- You'll be using the same markers.xml file you set up in the previous example.
- In the Mapplet Scratch Pad, click the "Go back to editor" button to display the scratch pad editor.
- Edit the Mapplet source in example_2_4_plot_remote_data.xml, and make sure the URL in the following line points to your markers.xml file:
var url = "http://yourserver.com/path/to/markers.xml"; - After editing example_2_4_plot_remote_data.xml, copy and paste the contents of the file into the Mapplet Scratch Pad.
- Click the Preview button in the scratch pad. You should see the Mapplet shown in Figure 2-4.
- Once the Mapplet loads, click the Load Data link to load the remote marker data. Once the band and venue data is loaded, you should see markers on the map as well as a list of band and venue links on the left side of your application. Click the band/venue links on the left, and notice how the appropriate info window appears.
Examining the Code
You have seen most of the code in this example, but a few new things are happening. Let's take a look at these now.
After you successfully load the data, you first check to make sure you have at least one marker in your XML file. If not, you show an error message:
if (concertData.length == 0) {
_gel("status").innerHTML = "Sorry, we don't have any concert data at this
time";
return;
}Once you know you have at least one marker in your XML data, you need to properly initialize a few map-related variables before you start placing markers on the map and updating your application. First, you create a new GLatLngBounds object bounds. Remember from the previous chapter that the bounds object is used to center the map based on all the points you plot on the map. Second, you call a function that you've created called cleanupMarkers(). This function removes event listeners added to the map markers. (You add event listeners to notify you when an event occurs, such as when a marker is clicked. You haven't added any markers to the map yet in the code, but it's coming next.) It's important to remove the event listeners in case the users click the Load Data button multiple times—you need to get in the habit of clearing up any markers and event listeners that you aren't using before you load new ones.
var bounds = new GLatLngBounds();
cleanupMarkers();
// function to remove all marker Event Listeners
function cleanupMarkers() {
if (markers == null)
return;
for (var i = 0; i < markers.length; i++) {
var marker = markers[i];
GEvent.clearListeners(marker, "click");
}
}Tip It's important to remove any event listeners that you aren't using anymore. This comes into account when you reload data or load new data sets from an application and are removing event listeners that aren't being used anymore. You can remove event listeners in a few ways, depending on how they were created: GEvent.removeListenerstner(), GEvent.clearListeners(), or GEvent.clearInstanceListeners(). Consult the Mapplet API reference for more details.
Before creating more markers, you reinitialize the array, markers, that stores all the GMarker objects you'll add to the map. You need to keep track of the markers so that you can display the proper marker info window when a band/venue link is clicked in your application. You'll learn more about this in a moment.
markers = new Array();Next, you loop through all the marker data found in the XML file that was just retrieved. The latitude and longitude is parsed from the XML and is used to create a GLatLng point. You then call the function createMarker() to create a new marker, passing in the band name (title) and venue name, which are used to decorate the marker info window. Once the marker is created, you place it on the map.
var point = new GLatLng(lat, lng);
var marker = createMarker(point, title + " - " + venue);
map.addOverlay(marker);After the marker has been placed on the map, you add it to your array, markers. You need a reference to each marker, because you'll need to display the marker's info window if a user happens to click a link in your application that corresponds with the marker.
markers[i] = marker;You add a link to the Mapplet's left panel of the application for each concert in the XML data. Notice that the link calls a function called clickMarker() with an argument of the marker index. The clickMarker() function helps you display the marker's info window when the link is clicked. This is key in making your Mapplet interactive, because the user can now either explore using the map or the list of concert links. Notice that in clickMarker() you use the GEvent.trigger() function to trigger the "click" action on the maker. This forces the "onclick" event on the marker, which in turn displays the info window for that marker.
displayHTML += "<div><a href='#' onclick='clickMarker(" + i + ");'>" +
title + " - " + venue + "</a></div>";
function clickMarker(index) {
GEvent.trigger(markers[index], "click");
}Finally, once you've created all the markers, you use the getBoundsZoomLevelAsync() function to help you scale the map so that you can see all the markers (two in this case) at the same time. You don't actually update the Mapplet application with your list of concert data until the map is properly centered and scaled.
Google's Data Cache
Google caches all data that is retrieved using the _IG_FetchContent() and _IG_FetchXmlContent() APIs. The data is cached to help speed up the retrieval of the data in your Mapplets and also to reduce load on the remote server (in case your Mapplet gets a lot of users at once). If the content you are fetching is refreshed more than once an hour, you need to use the refreshInterval parameter to bypass the cache and fetch the latest content from the remote server. You can use this parameter, which is measured in seconds, to specify the interval at which you want to refresh the cached data. You'll use this parameter in your applications in the next chapter, but here's what it looks like if you need to refresh your data every 10 minutes:
_IG_FetchXMLContent("http://yourserver.com/path/to/markers.xml",
callback, { refreshInterval: (60 * 10) });Note If you do use the refreshInterval to bypass the cache, Google recommends you use a range of 60 (one minute) to 3600 seconds (one hour).
Summary
Nice job. You've created an interactive map using data from a remote server! I hope you are starting to see the power of combining remote data feeds and services with a map. You're just getting started.
Over the next few chapters, you'll start mapping live data from real web services. You now have the tools to start creating your mashup. Are you ready? Let's move on....