Other people are using your site, and you don’t know it. They may be doing so as part of a mashup. They may be running search engine crawlers to index your content. Or they may be competitors checking up on you. Whatever the case, you need to track and monitor them.
Your site may offer formal web services or Application Programming Interfaces (APIs) to let your users access your application programmatically through automated scripts. Most sites have at least one trivially simple web service—the RSS feed, which is simply an XML file retrieved via HTTP.
Most real web services are more “heavyweight” than syndication feeds, but even the most basic web services need to be monitored. If letting people extend your web application with their own code is important to your business, monitor the APIs to ensure they are reliable and that people are using them in appropriate ways.
Some RSS management tools, like FeedBurner, will showcase “unusual” uses of your feed automatically—for example, someone who is pulling an RSS feed into Yahoo! Pipes for postprocessing, as shown in Figure 3-20.
Figure 3-20. FeedBurner’s Uncommon Uses report can show you ways in which others are using your RSS feed
If your web services include terms of service that limit how much someone can use them, track offending users before a greedy third-party developer breaks your site. Note that many analytics packages that rely on JavaScript can’t be used to track API calls because there’s no way to reliably embed JavaScript in the content they deliver.
What to watch: Traffic volume and number of requests for each API you offer; number of failed authentications to the API; number of API requests by developer; top URLs by traffic and request volume.
Your site’s data can easily appear online in a mashup. By combining several sites and services, web users can create a new application, often without the original sites knowing it. Tools like Yahoo! Pipes or Microsoft’s Popfly make it easy to assemble several services without any programming knowledge.
Some mashups are well intentioned, even encouraged. Google makes it easy for developers to embed and augment their maps in third-party sites. Other repurposing of content, such as republishing blog posts or embedding photos from third-party sites, may not be so innocent. If someone else reposts your content, your search ranking goes down, and if someone else embeds links to media you’re hosting, you pay for the bandwidth.
If this is happening to you, you’ll see referring URLs belonging to the mashup page, and you can track back to that URL to determine where the traffic is coming from and take action if needed. You may have to look in server logs or on a sniffer, because if the mashup is pulling in a component like a video or an image, you won’t see any sign of it in JavaScript-based monitoring.
Note
Although the term “Sniffer” is a registered trademark of Network General Corporation, it is also used by many networking professionals to refer to packet capture devices in general.
Try to treats mashups as business opportunities, not threats. If you have interesting content, find a way to deliver it that benefits both you and the mashup site.
What to watch: URLs with a high volume of requests for an object on your site without the normal entry path; search results containing your unique content.
Some SaaS applications may connect to their subscribers’ enterprise software through dedicated links between the subscriber’s data center and the SaaS application in order to exchange customer, employee, and financial data. Relying on these kinds of third-party services can affect the performance and availability of your website. Travel search sites, for example, are highly dependent on airline and hotel booking systems for their search results.
While not directly related to web monitoring, the performance of these third-party connections must be tested and reported as part of an SLA. Excessive use of enterprise APIs or long delays from third-party services may degrade the performance of the website. Keep nonweb activity in mind when monitoring the web-facing side of the business, and find ways to track private API calls to enterprise clients.
What to watch: Volume and performance of API calls between the application and enterprise customers or data partners.