In this chapter, we will begin to explore how search engines work. Building a strong foundation on this topic is essential to understanding the SEO practitioner’s craft.

As we discussed in Chapter 1, people have become accustomed to receiving nearly instantaneous answers from search engines after they have submitted a search query. In Chapter 1 we also discussed the volume of queries (more than 4,500 per second), and Google reported in July 2008 that it knew of about 1 trillion pages on the Web (http://googleblog.blogspot.com/2008/07/we-knew-web-was-big.html).

Underlying this enormous data processing task is the complex nature of the task itself. One of the most important things to understand about search engines is that the crawlers (or spiders) used to visit all the web pages across the Web are software programs. Software programs are only as smart as the algorithms used in implementing them, and although artificial intelligence is being increasingly used in those algorithms, web crawling programs still don’t have the adaptive intelligence of human beings.

Software programs cannot adequately interpret each of the various types of data that humans can—videos and images, for example, are to a certain extent less readable by a search engine crawler than they are through the eyes of humans. In the section Vertical Search Engines, we will discuss how advances in image and video search have enabled search engines to inch even closer to human-like understanding.