To augment Google’s link: command, you can utilize Yahoo!’s Site Explorer tool, which can provide similar functionality. Although it is still possible to write a scraper script to pick up Yahoo!’s results, being able to get a clean text file may be more than sufficient for smaller sites.

With Yahoo! Site Explorer, you can export the first 1,000 backlinks to a file in Tab Separated Value (TSV) format just by clicking on the TSV link, as shown in Figure 5-1.

Figure 5-1. Yahoo! Site Explorer

After downloading the TSV file, you can parse the 1,000 links to see the link anchor text in each destination page. You can also sign up with Yahoo! to get its API to extract the same data.

List of inlinks to the site oreilly.com
TITLE                        URL                       SIZE    FORMAT
Apache Software Foundation   http://www.apache.org/    22809   text/html
Slashdot                     http://slashdot.org/      144476  text/html
...

perl inlinksAnalysis.pl oreilly.com_inlinks.txt oreilly.com

http://radar.oreilly.com/, 260, "O'Reilly Radar"
http://www.oreilly.com/, 108, "O'Reilly Media"
http://www.oreilly.com, 105, "Oreilly Logo"
http://oreilly.com/, 103, "O'Reilly Media"
http://jobs.oreilly.com/, 69, "Jobs"
http://press.oreilly.com/, 68, "Press Room"
http://en.oreilly.com/webexny2008/public/content/home, 66, "Web 2.0 Expo"
http://toc.oreilly.com/, 66, "Tools of Change for Publishing"
http://academic.oreilly.com/, 66, "Academic Solutions"
http://oreilly.com/feeds/, 66, "RSS Feeds"
http://oreilly.com/contact.html, 66, "Contacts"
http://oreilly.com/terms/, 66, "Terms of Service"
http://oreilly.com/oreilly/author/intro.csp, 66, "Writing for O'Reilly"
http://digitalmedia.oreilly.com/, 65, "Digital Media"
http://ignite.oreilly.com/, 64, "Ignite"
. . .

How fast are you accumulating backlinks? This can become a negative ranking factor if your site is found to be using an unethical linking scheme.

If all your backlinks are using the same anchor text, search engines may view this as suspicious activity, which might trigger a manual inspection by the search engines and a delisting of your site. Let’s say your site had 20 backlinks yesterday, and today you suddenly have more than 100,000. Wouldn’t that look suspicious?

It is important for backlinks to contain relevant anchor text. Here are some examples of good and bad anchor text:

<!--Good-->
<a target=_new href="http://yoursite.com/basketball>Learn about
Basketball</a>
<a target=_new href="http://yoursite.com/basketball>Talk about
Basketball at www.yoursite.com</a>

<--Bad-->
<a target=_new href="http://yoursite.com/basketball">Click Here</a>
<a target=_new href="http://yoursite.com/basketball">
http://yoursite.com</a>
<a target=_new href="http://yoursite.com/basketball">more..</a>
<a href="http://yoursite.com/basketball"><img
src="http://www.somesite.net/pic1.jpg"></a>

The good examples use relevant text in the links. The bad examples, although they use the same destination URL, do not use relevant text. Ideally, the referring sites should also be talking about the same subject matter. At the very least, the referring page should be on the same topic.

For good or bad, search engines have given certain websites a biased advantage over all other sites for most of the popular keywords. The basis of this practice stems from a popular ranking algorithm developed at the University of Toronto, called the Hilltop Algorithm.

The creators of the Hilltop Algorithm define it as follows:

At the time of this writing, in a search for the keyword hilltop algorithm, Google’s search results look like Figure 5-2.

Figure 5-2. Biased SERP

In this example, the Wikipedia search result came in ahead on Google, as it did on Yahoo! and Bing. Now, if we open the Wikipedia result, we don’t really see much except for the definition. So a single-paragraph page with a ton of useless Wikipedia links gets a PageRank value of 5 and the first spot on Google. Meanwhile, all other search results on the first page contain much more information on the given keywords.

Google at least showed one page from the University of Toronto in its top 10 results—unlike Yahoo! and Bing. This is because the Wikipedia page contains the exact same link from the University of Toronto!

This should speak volumes about the importance of backlinks from expert sites. The pattern of Wikipedia importance is repeated countless times, so get a link on Wikipedia if you can!

When Matt Cutts (Google search quality engineer) was asked about the extra ranking boost from inbound links originating from .edu and .gov TLDs, he responded as follows:

Although Google denies that .edu and .gov links get a PageRank boost, you can easily construe that the domain extension does matter in some sense. When Google flat out denies something, it is almost always because it is not true. But when Google is cagey about something, you need to beware. Most people believe that .edu and .gov links are more valuable than others because those sites typically have strong PageRank and therefore more link juice. It is doubtful that Google actually gives these sites greater weight simply because of the TLD extension. When your site has a link from an .edu or .gov link, it says a great deal about the site’s trust and authority.

It is very hard to get an .edu or .gov domain. Only one registrar today has permission to grant .edu domains: EDUCAUSE, a nonprofit organization that won the right to be the sole registrar of all .edu domains in 2001. Not everyone can get an .edu domain, and .edu domains are not transferable. Eligibility requirements are very strict (http://net.educause.edu/edudomain/):

The situation is similar for .gov TLDs; these domains can be registered only at http://www.dotgov.gov (see http://bit.ly/1givpb):

Attaining links from TLDs such as .edu and .gov is valuable provided that the referring page is on the same topic as the recipient page. Historically, very little spam has originated from these domains. The added value of these links is that they tend to change very little over time, and age is an important off-page ranking factor, as we’ll discuss later in this chapter.

In some countries, there are strict regulations in terms of domain name assignments for ccTLDs. Country-specific TLDs, especially expansive TLDs, are believed to carry more clout, as spammy sites are usually those of cheap TLDs such as .com or .info.

Other places where domain extensions matter are in localized searches. The benefit of country-specific TLDs can also be augmented by ensuring that web servers hosting the domain are using IPs from the same region. If the question is whether to pursue getting some inbound links from these foreign TLDs, the answer is almost always yes.

Traditionally, specialized web directories carried more authority than regular sites. Google and others have relied on directories not just in terms of ranking factors, but also in using directory descriptions as their own descriptions in search results. Directories are considered similar to other “expert” sites such as Wikipedia.

Be careful which directory you choose to use for your site, as not all are created equal. Check the PageRank of each directory to determine whether it is worth having your listing there. The higher the PageRank is, the better. Also ensure that these directories are passing link juice.

The age of the backlinks conveys a message of trust, and therefore a perceived link quality. A new backlink on a page that is constantly changing is not as beneficial as an old backlink on the same page.

In some cases, new backlinks exhibit a state of “purgatory” before they attain trust. In other words, the real benefit of new backlinks takes time to propagate. You have to start somewhere, so build your links for their future benefits.

Backlinks found closer to the top of an HTML page are perceived as being more important than those found farther down the page.

This makes sense from a website usability perspective as well. Links found on long scrolling pages are much more likely to do better if they are found at the top of the page.

Be careful with this, though. With some simple CSS, links rendered at the top of a page in a typical web browser can easily be placed at the bottom of the actual (physical) HTML file.

Your backlinks may not be the only backlinks on a particular referring site. The greater the number of other links found on the same page, the less of a benefit you will incur. The incoming PageRank is split among all other outbound links found on that page.

If your site is one of many on the same subject, its relative popularity among all the other peer sites comes into play. Even if you feel your content or product is better than your competition, your rankings may be suffering due to a lack of relative popularity.

You can run linkchecker.pl with the following code:

> perl linkchecker.pl http://www.somedomain.com > somedomain.csv

Depending on the size of the site, it can take anywhere from a few seconds to several hours to check the site for broken links. If you have a really large site, you may want to partition this activity into several smaller checks. After the script is finished, you can open the report file in Excel. Figure 5-3 shows sample output.

Figure 5-3. Link checker report

The spreadsheet in Figure 5-3 shows the following columns:

Simply inspect the spreadsheet for 404 status codes to find your broken links, whether they happen to be internal or external. You may also want to inspect 500 status codes, as they indicate server errors.

We can learn a lot about a visitor as soon as he arrives on our home page. First, we know his IP address. Using his IP address we can pinpoint his geographic location. Based on his geographic location, we can create a customized index based on the patterns of previous visitors from the same geographic location.

We can also record cases in which another link referred the visitor to our home page. Depending on his Internet service provider and his assigned IP range, we can be so precise that we can pinpoint his city or town location!

In addition to IP and referrer values, we can also learn a great deal of additional information, including browser version, associated (installed) browser software, operating system version, current screen resolution, whether the visitor’s browser cookies are enabled, the speed of his Internet connection, and more. That’s a lot of information already, and the visitor has not even clicked on the Search button!

Once the visitor clicks on the Search button, things start to get more interesting. First, the visitor is telling the search engine what he is looking for. The search engine responds with a results page based on the current index for the specific region the visitor is coming from. Between the time that the results page shows up and the time the visitor clicks on a specific search result, many things can be recorded, including:

As you can see, we can learn many things from a single visitor. But the picture becomes much clearer when we combine these stats with those from thousands or even millions of other visitors. With more data available, it becomes easier to spot patterns. Just using a small sample of visitor data is often enough to approximate which links are popular and which are not so useful.

For example, say that for a particular keyword, most visitors clicked on the third result, immediately followed by the sixth result. This could indicate that the third result did not contain good or sufficient information for this keyword, as visitors kept coming back to inspect the sixth result. It could also say that other results shown on the SERPs were not enticing enough for visitors to click on.

As another example, say that visitors clicked on the first result and then came back 10 minutes later to click on the fourth result. This could signify that visitors liked the information in the first result but chose to research further by clicking on the fourth result.

Now let’s consider CTR, which is the number of times a specific result is clicked on in relation to the total number of times it is shown. A high CTR is something all webmasters want for their URLs. A low CTR could signal to search engines that the result is not interesting or relevant.

Search engines can also gather information on multiple search queries, especially if they were all performed through the same browser window in the same session. These sorts of search queries can indicate keyword correlation. Although such information is already empowering search engines, it is not perfect. There are ways to make it better or more accurate.

If you are on a dedicated server, you may already have access to website monitoring tools and insight into your website performance. If your website is on a shared host, chances are that all the websites sharing that host are competing for a slice of the available CPU pie.

Every serious website owner needs to proactively monitor his website’s performance, as poor performance can have adverse web crawling and indexing effects. You can monitor a website with a relatively simple Perl script. One such script is mymonitor.pl, which you can find in Appendix A. Running the script is as simple as:

perl mymonitor.pl http://www.somedomain.net/somelink.html

The script records the elapsed time it takes to download the supplied URL while writing to several text files. Each line of these text files is formatted in the following way:

mm-dd-yyyy hh:mm:ss;10;15;20;[actual elapsed time]

Example text file content looks like this:

12-6-2009 6:0:4;10;15;20;3.25447
12-6-2009 6:5:3;10;15;20;1.654549
12-6-2009 6:10:2;10;15;20;1.765499
12-6-2009 6:15:9;10;15;20;8.2816

Numbers 10, 15, and 20 represent markers in seconds for easier viewing on a chart. Several free graphics packages are available on the market. Plotting our data with amCharts’ charting Flash demo version software yields a graph similar to Figure 5-4.

Figure 5-4. Plotting performance with amCharts.com

Figure 5-4 illustrates degraded performance from about 6:00 a.m. to 7:30 a.m. Knowing such information is valuable, as it can provide clues for further analysis. The last thing left to do is to automate the process of running the script. If you are on a Linux system, you can simply set up a crontab job. To run the script every five minutes, you could set up your crontab as follows:

#Run test every 5 minutes

*/5 6-22 * * * perl mymonitor.pl http://www.domain.net/link1.html

The preceding crontab entry would run every five minutes between 7:00 a.m. and 11:00 p.m. every day. This is only one link, and nobody is preventing you from setting up several links to track. Also note that you could use the Windows Task Scheduler to set up the script-running schedule.

A longer registration period suggests a serious domain owner, whereas short registrations such as single-year registrations could be interpreted as suspicious. This is of no surprise, as most search engine spam tends to come from short-lived domains.

Google recently confirmed that expired domains, when renewed, are essentially treated as new domains. In other words, if an expired domain had a high PageRank, that PageRank will be wasted if the domain is allowed to expire before someone else buys it.

Domains that have existed for a long time carry an age trust factor. New pages added to these domains tend to get higher rankings more quickly. As we discussed, older domains convey authority and are given much more leeway than newer ones. If your domain is new and relevant, focus on the daily activities of your business while building compelling and relevant content—and plan for the long term.