Say you’ve just purchased your domain name. Unless you already changed the default DNS server assignments, chances are that when you type in your domain name, you get to a domain parking page served by your domain registrar. It can be somewhat annoying to see the domain registrar’s advertisements plastered all over your domain while passing (at least temporarily) your domain’s link juice (if any) to its sites.

Most people in this situation will put up an “Under Construction” page or something similar. If that is the case, you really do not want search engines to index this page. So, in your index.html (or equivalent) file, add the following robots meta tag:

<meta name="robots" content="noindex">

The suggested practice is to have a “Coming Soon” page outlining what your site will be all about. This will at least give your visitors some ideas about what to expect from your site in the near future. If for some reason you want to block crawling of your entire site, you can simply create a robots.txt file in the root web folder:

User-agent: *
Disallow: /

The star character (*) implies all web spiders. The trailing slash character (/) signifies everything after the base URL or domain name, including the default document (such as index.html).

Content duplication is a common issue in SEO. When your site is serving the same content via different link combinations, it can end up splitting your link juice across the different link permutations for the page.

The robots.txt file can be helpful in blocking the crawling of duplicated content, as we will discuss in some examples later in this chapter. We will discuss content duplication in detail in Chapter 14.

Hiding specific pages or files from the SERPs can be helpful. When it comes to documents that should be accessed by only authenticated clients, REP falls short in providing that sort of functionality. Using some sort of authentication system is required. One such system is the .htaccess (Basic authentication) method.

Protecting directories with .htaccess

Implementing .htaccess protection is relatively straightforward—if you are running your site on an Apache web server on a Linux-flavored OS. Let’s look at an example. Suppose we have a directory structure as follows:

.public_html
.public_htmlimages
.public_htmllog
.public_htmlprivate

We are interested in using Basic HTTP authentication for the private subdirectory. Using Basic HTTP authentication requires the creation of two text files, namely .htaccess and .htpasswd. Here is how the .htaccess file might look:

AuthType Basic
AuthName "Restricted Area"
AuthUserFile "/home/nonpublicfolder/.passwd"
require valid-user

AuthType corresponds to the authentication type, which in our case is Basic HTTP authentication. AuthName corresponds to a string used in the authentication screen after the “The site says:” part, as shown in Figure 9-1. AuthUserFile corresponds to the location of the password file.

Figure 9-1. Basic HTTP authentication

The .passwd file might look as follows:

guest:uQbLtt/C1yQXY
john:l1Ji97iyY2Zyc
jane:daV/8w4ZiSEf.
mike:DkSWPG/1SuYa6
tom:yruheluOelUrg

The .passwd file is a colon-separated combination of username and password strings. Each user account is described in a separate line. The passwords are encrypted. You can obtain the encrypted values in several ways. Typically, you would use the following shell command:

htpasswd -c .htpasswd guest

Upon executing this command, you are asked to enter the password for user guest. Finally, the .htpasswd text file is created. On subsequent additions of users, simply omit the -c command-line argument to append another user to the existing .htpasswd file. You can also create encrypted passwords by using the online service available at the Dynamic Drive website (http://tools.dynamicdrive.com/password/).

All sites need to perform maintenance at some point. During that time, Googlebot and others might try to crawl your site. This begs the question: should something be done to present spiders with a meaningful message as opposed to a generic maintenance page? The last thing we want is for search engines to index our maintenance page.

The most appropriate response is to issue the HTTP 503 header. The HTTP 503 response code signifies service unavailability. According to the W3C, the HTTP 503 code is defined as follows (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html):

Handling this scenario is easy with some PHP code. Using the PHP header method, we can write the appropriate headers. Here is how the code might look:

<?php
ob_start();
header('HTTP/1.1 503 Service Temporarily Unavailable'),
header('Status: 503 Service Temporarily Unavailable'),
header('Retry-After: 7200'),
?><html>
<head>
<title>503 Service Temporarily Unavailable</title>
</head>

<body>
<strong>Service Temporarily Unavailable</strong><br><br>
We are currently performing system upgrades. Website is not available
until 5am. <br>Please try again later and we apologize for any
inconvenience.
<!--file:maintenance.php-->
</body>
</html>

The sample code accomplishes two things: it communicates the service outage to the user while also letting spiders know they should not crawl the site during this time. Please note that for this solution to fully work you would also need to write a rewrite rule that would be used to forward all domain requests to this page. You can do this with a simple .htaccess file, which you should place in the root folder of your website. Here is how the file might look:

Options +FollowSymlinks

RewriteEngine on
RewriteCond %{REQUEST_URI} !/maintenance.php$
RewriteCond %{REMOTE_HOST} !^57.168.228.28

RewriteRule $ /maintenance.php [R=302,L]

The .htaccess file also allows you as the administrator to browse the site from your hypothetical IP address (57.168.228.28) during the upgrade. Once the upgrade is complete, the maintenance .htaccess file should be renamed (or removed) while restoring the original .htaccess file (if any).

Anyone hitting the site during the maintenance period would see the following HTTP headers, as produced by running the maintenance.php script:

GET /private/test.php HTTP/1.1
Host: www.seowarrior.net
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7)
Gecko/2009021910 Firefox/3.0.7
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cookie: PHPSESSID=308252d57a74d86d36a7fde552ff2a7f

HTTP/1.x 503 Service Temporarily Unavailable
Date: Sun, 22 Mar 2009 02:49:30 GMT
Server: Apache/1.3.41 (Unix) Sun-ONE-ASP/4.0.3 Resin/3.1.6
mod_fastcgi/2.4.6 mod_log_bytes/1.2 mod_bwlimited/1.4
mod_auth_passthrough/1.8 FrontPage/5.0.2.2635 mod_ssl/2.8.31
OpenSSL/0.9.7a
Retry-After: 7200
X-Powered-By: PHP/5.2.6
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html

The boldface strings are the string values we specified in the PHP file. The first string represents the HTTP status code (503). The second string is the Retry-After segment representing a note to the web crawler to try crawling the site again in two hours (7,200 seconds).

Although website bandwidth is relatively inexpensive today when compared to the early days of the mainstream Internet, it still adds up in terms of cost. This is especially the case with large websites. If you are hosting lots of media content, including images, audio, and video, you may want to prohibit crawlers from accessing this content. In many cases, you may also want to prohibit crawlers from indexing your CSS, JavaScript, and other types of files.

Before implementing any crawler blocking, first check with your web logs for web spider activities to see where your site is taking hits. Keep in mind that anything you block from crawling will potentially take away some traffic. For example, all the big search engines provide specialized search for images. Sometimes people can learn about your site when only searching for images.

Certain web pages may be hogging your website’s CPU cycles. Taking additional hits on these pages during peak utilization hours can further degrade your site’s performance, which your visitors will surely notice.

Sometimes you have no choice but to ensure that crawlers can see these pages. You can utilize several different performance optimizations. These include the use of web page caching, web server compression, content compression (images, video, etc.), and many others techniques. Sometimes you have no other choice but to limit crawling activities by preventing crawl access to CPU-intensive pages.

The Allow directive tells web crawlers that the specified resources can be crawled. When multiple directives are applicable to the same URL, the longest expression takes precedence. Suppose we had a robots.txt file as follows:

1 User-agent: *
2 Disallow: /private/
3 Allow: /private/abc/

We have one Disallow directive for the private directory. In addition, we have one Allow directive for the abc subdirectory. Googlebot comes along and has to decide which URL references it needs to crawl:

The answer is that Googlebot needs to crawl all but the second URL, since the longest directive it matches is the Disallow directive on line 2 of the preceding code. The first URL is a perfect match of the Allow directive. The third URL also matches the Allow directive. Finally, the last URL is allowed, since it does not match the Disallow directive.

When in doubt, always ask the following question: what is the longest directive that can be applied to a given URL? As a rule of thumb, if no directive can be applied to a given URL, the URL would be allowed.

Let’s examine another example. What happens if you have an empty Allow directive? Here is the code fragment:

User-agent: *
Allow:

The simple answer is that nothing happens. It makes no sense to place Allow directives when there is not a single Disallow directive. In this case, all documents would be allowed for crawling.

The Disallow directive was the original directive created. It signified the webmaster’s desire to prohibit web crawler(s) from crawling specified directories or files. Many sites choose to use this directive only. Its basic format is:

Disallow: /directory/
Disallow: /file.ext

For the sake of completeness, let’s see what happens if we have an empty Disallow directive. Consider the following code fragment:

User-agent: *
Disallow:

In this case, all documents would be allowed for crawling. This seems counterintuitive. But let’s think for a moment. Yes, the Disallow directive is used, but it does not have any arguments. Since we did not specify any file (or directory) to disallow, everything would be allowed. Let’s look at another example:

User-agent: *
Disallow: /

In this case, everything would be disallowed for crawling. The forward slash (Disallow) argument signifies the web root of your site. What a difference a single character makes! Note that both Allow and Disallow can employ wildcard characters.

Two wildcards are used in robots.txt: $ and *. Both have functionality similar to how they’re used in Perl regular expression matching. You use the dollar sign wildcard ($) when you need to match everything from the end of the URL. You use the star wildcard character (*) to match zero or more characters in a sequence. Let’s look at some examples in the following fragment:

1 User-agent: *
2 Disallow: *.gif$
3
4 User-Agent: Googlebot-Image
5 Disallow: *.jpg$
6
7 User-Agent: googlebot
8 Disallow: /*sessionid

The catchall block (the first two lines) prohibits all obeying spiders from crawling all GIF images. The second block tells Google’s image crawler not to crawl any JPG files. The last block speaks to Googlebot. In this case, Googlebot should not crawl any URLs that contain the sessionid string. As an example, suppose we have Googlebot and Googlebot-Image crawlers deciding what to do with the following link references:

The net result would be that the Google-Image crawler would not crawl header.jpg while crawling header.gif. The GIF image would be crawled because the Google-Image crawler ignores the catchall crawler segment when it finds the Google-Image segment. Furthermore, Googlebot would not crawl the second URL, which contains a match for the Disallow directive on line 8.

The big three search engines collaborated on the introduction of the Sitemap directive. Here is an excerpt from Yahoo! Search Blog:

The Sitemap location directive simply tells the crawler where your Sitemap can be found. Here is an example of the robots.txt file utilizing the Sitemap directive:

Sitemap: http://mydomain.com/sitemap.xml

User-Agent: *
Disallow:

The location of the Sitemap directive is not mandated. It can be anywhere within the robots.txt file. For full coverage of the Sitemaps protocol, see Chapter 10.

Only Yahoo! and Bing use the Crawl-delay directive. They use it to tell Slurp and MSNBot crawlers how frequently they should check new content. Here is an example:

Sitemap: /sitemap.xml

User-agent: *
Disallow: /private/

User-agent: Slurp
Disallow:
Crawl-delay: 0.5

User-agent: MSNBot
Disallow:
Crawl-delay: 15

In this example, we have three crawler blocks in addition to the Sitemap directive placed at the beginning of robots.txt. The first crawler block is of no interest to us, as it is the catchall crawler. The second block tells the Slurp crawler to use a Crawl-delay value of 0.5. The last block instructs the MSNBot crawler to use a Crawl-delay of 15.

The values of the Crawl-delay directive have different meanings to Slurp and MSNBot. For MSNBot, it represents the number of seconds between each crawl. The current allowed range is between one and 600 seconds.

Slurp interprets the Crawl-delay directive a bit differently. Here is what Yahoo! says:

Although Google does not support the Crawl-delay directive via robots.txt, it does have its own alternative. Under the Settings section of Google Webmaster Tools, you can set up the specific crawl rate for your site, as shown in Figure 9-2.

Figure 9-2. Setting the site’s crawl rate in Google Webmaster Tools

You do not have to set up Google’s crawl rate unless you are experiencing performance degradations attributed to crawler activities. The same is true of the Crawl-delay directive.

You can disallow image crawling in a few ways. These include prohibiting specific image crawlers, prohibiting specific image folders, and prohibiting specific file extensions.

Let’s suppose we want to prohibit all image crawling for the most popular search engines. We also want to accomplish this on the sitewide level. The following fragment shows how we can do this:

User-agent: Yahoo-MMCrawler
Disallow: /

User-agent: msnbot-media
Disallow: /

User-Agent: Googlebot-Image
Disallow: /

In this example, all other crawlers would be able to crawl your images. In some situations you might not care about other crawlers. You can extend this example by explicitly using folder names.

You should always store your images in centralized directories to allow for easy maintenance, and to make it easier to block crawlers. The following fragment speaks to the major crawlers in addition to any other crawlers obeying REP:

User-agent: Yahoo-MMCrawler
Disallow: /images/
Allow: /images/public/

User-agent: msnbot-media
Disallow: /images/
Allow: /images/public/

User-Agent: Googlebot-Image
Disallow: /images/
Allow: /images/public/

User-Agent: *
Disallow: /images/

In this example, we are blocking all crawlers from crawling all images in the images folder, with one exception. Google, Yahoo!, and Bing are allowed to crawl any images that contain /images/public/ in their URL path.

The easiest way to block all images is by using robots.txt wildcards. Suppose you are using only three image types: GIF, PNG, and JPG. To block crawling of these types of images, you would simply use the following code:

User-agent: *
Disallow: /*.gif$
Disallow: /*.jpg$
Disallow:/*.png$

If your target audience is in North America or Europe, you may want to target only Google and Yahoo!. The following fragment shows how to implement this sort of setup:

User-agent: slurp
Disallow:

User-agent: googlebot
Disallow:

User-agent: *
Disallow: /

You already saw examples of blocking specific image file types. Nothing is restricting you from blocking the crawling of many other file types. For example, to restrict access to Microsoft Office files, simply use the following fragment:

User-agent: *
Disallow: /*.doc$
Disallow: /*.xls$
Disallow: /*.ppt$
Disallow: /*.mpp
Disallow: /*.mdb$

Did you know that the Internet Archive website contains billions of web pages from the mid-1990s onward? For archiving purposes, the Internet Archiver makes use of its crawler, which is called ia_archiver. Most website owners are not even aware of this. Sometimes this is something you may not wish to allow. To block your site from being archived, use the following fragment:

User-agent: ia_archiver
Disallow: /

It is perfectly acceptable to combine HTML meta directives into a single meta tag. For example, you may want to noindex and nofollow a particular page. You would simply list both directives, separated by a comma, as in the following example:

<meta name="robots" content="noindex,nofollow" />

Let’s say you have a page that is not your canonical (preferred) page, but is being linked to by some highly trusted site. In this case, you may want to do the following:

<meta name="robots" content="noindex,follow" />

In this example, crawlers can follow any outbound links away from this page, as you might feel that some of these links are your canonical links (such as your home page).

You can define HTML meta tags to target specific spiders. For example, to instruct Googlebot and Slurp not to index a particular page, you could write:

<meta name="googlebot" content="noindex" />
<meta name="slurp" content="noindex" />

Yahoo! uses a couple of extra directives, namely noydir and robots-nocontent. The first directive is used to instruct Yahoo! not to use Yahoo! directory descriptions for its search engine results pertaining to this page. Here is the basic format:

<meta name="robots" content="noydir" />

The second directive, robots-nocontent, operates on other HTML tags. It is not to be used with meta tags. Utilizing this tag allows you to prevent certain portions of your page, such as navigational menus, from being considered by Yahoo!’s indexing algorithms. The net effect of this method is increased page copy relevance (higher keyword density). Here are some examples:

<div class="robots-nocontent"><!--menu html--></div>

<span class="robots-nocontent"><!--footer html--></span>

<p class="robots-nocontent"><!-- ads html--></p>

These examples should be self-explanatory. Marking text that is unrelated to the basic theme or topic of this page would be advantageous. Note that there are other ways of doing the same thing for all search engines.

One such way is by loading ads in iframes. The actual ads would be stored in external files loaded by iframes at page load time. Crawlers typically ignore iframes. You could also use Ajax to achieve the same effect.

There are three Google-specific directives: unavailable_after, noimageindex, and notranslate. The first directive, unavailable_after, instructs Google to remove the page from its index after the date and time expiry. Here is an example:

<meta name="googlebot" content="unavailable_after: 12-Sep-2009
12:00:00 PST">

In this example, Googlebot is instructed to remove the page from its index after September 12, 2009, 12:00:00 Pacific Standard Time. The second directive, noimageindex, instructs Google not to show the page as a referring page for any images that show up on Google image SERPs. Here is the format:

<meta name="googlebot" content="noimageindex">

The third directive, notranslate, instructs Google not to translate the page or specific page elements. It has two formats. The following fragment illustrates the meta format:

<meta name="google" value="notranslate">

Sometimes it is useful to translate only certain parts of a page. The way to accomplish that is by identifying page elements that are not to be translated. The following fragment illustrates how you can do this:

<span class="notranslate">Company Name, Location</span>
<p class="notranslate">Brand X Slogan</p>