3.1. Robots Deliver
We're going to start with the basics of how the search engines work, and a major component of this is a robot, or spider, which is software that gathers information about your site and brings it back to be analyzed by a powerful central "engine." This activity is referred to as crawling or spidering. There are lots of different metaphors for how robots work, but we think ants make the best one. Think of a search engine robot as an explorer ant, leaving the colony with one thought on its mind: Find food. In this case, the "food" is HTML text, preferably lots of it, and to find it, the ant needs to travel along easy, obstacle-free paths: HTML links. Following these paths, the ant (search engine robot), with insect-like single-mindedness, carries the food (text) back to its colony and stores it in its anthill (search engine database). Thousands and thousands of the little guys are exploring and gathering simultaneously all over the Internet. (See Figure 3.1 for a visual example.) If a path is absent or blocked, the ant gives up and goes somewhere else. If there's no food, the ant brings nothing back.
Figure 3.1. Search engine robots at work
So basically, when you think of a search engine, consider it a database that holds pieces of text that have been gathered from millions of sites all over the Web.
What sets that engine in motion? A search. When a web surfer enters the term "grape bubble gum" into the search engine, all of the sites that might be relevant for that term are brought to the forefront. The search engine sifts through its database for sites containing terms like "grape growers," "stock market bubble," and "gum disease." It uses a secret formula—a.k.a. a search ranking algorithm—to sort the results, and in a fraction of a second, a list of relevant sites, many containing the exact phrase "grape bubble gum," (or with links from other sites containing this phrase) will be returned in the results page.
There are lots of things that factor into the way robot search engines determine the rank for their main search results. But just for a start, in order to be in the running for ranks, you need to provide HTML text to feed the search engines and HTML links as clear paths to the food. Keeping those robots well fed and happy is one of your biggest priorities.
A Search Engine by Any Other NameWe like to talk about search engines at parties—hey, it's how we get our clients—and when we use the term search engines with our pals, we mean search sites like Google, Yahoo!, and MSN. But folks in the search industry can get pretty picky about terminology. Technically, a search engine is the software that is used to retrieve information from an indexing database, while a search site is a website that combines and displays all of that information, often from multiple sources. But, frankly, we don't give a hoot about the technical accuracy of the term. Everyone and his mother calls Yahoo! a search engine, and—with apologies to the purists out there—we do, too. |