Chapter 2
Meeting the Search Engines
In This Chapter
- Finding common threads among the engines
- Meeting the major and minor search engines
- Finding your niche in the vertical engines
- Understanding metasearch engines
All search engines try to make their results the most relevant. They want to make you happy, because when you get what you want, you’re more likely to use that search engine again. The more you use them, the more money they make. It’s a win/win situation. So when you do your search on classic car customization and find what you’re looking for right away instead of having to click through five different pages, you’ll probably come back and use the same search engine again.
In this chapter, you meet the major search engines and discover their similarities and differences, get familiar with the difference between organic and paid results, and gain a better understanding of how the search engines get their organic results. Plus, you find out about the search engines’ paid search programs and get help deciding whether metasearch engines have a place in your SEO campaign.
Finding the Common Threads among the Engines
To keep their results relevant, all search engines need to understand the main subject of a website. You can help the search engines find your website by keeping in mind the three major factors they’re looking for:
- Content: Content is the meat and bones of your website. It’s all the information your website contains, not just the words but also the Engagement Objects (the images, videos, audio, interactive technologies, and so on that make up the visual space). Your page's relevancy increases based upon your perceived expertise. And expertise is based on useful, keyword-containing content. The spiders, the software the search engines use to read your website, also measure whether you have enough content that suggests you know what it is you’re talking about. A website with ten pages of content is going to rank lower than a website with ten thousand pages of content, assuming that they are equally relevant.
- Popularity: The Internet is a little like high school in that you are popular as long as a lot of people know you exist and are talking about you. Search engine spiders are looking for how many people are linking to your website, along with the number of outgoing links you have on your own site. Google really loves this factor.
- Architecture: If you walk into a grocery store and find everything stacked haphazardly on the shelves, it’s going to be harder to find things, and you might just give up and go to another store that’s better organized. Spiders do the same thing. As we mention in Chapter 1 of this minibook, search engines love Wikipedia because of how it’s built. It’s full of searchable text, Alt attribute text, and keyword-containing hyperlinks that support terms used on the page.
You also have some control over two variables that search engines are looking at when they set the spiders on you. One is your site’s response time, which is how fast your server is and how long it takes to load a page. If you’re on a server that loads one page per second, the bots request pages at a very slow rate. A second seems fast to us, but it's an eternity for a bot that wants five to seven pages per second. If the server can't handle one page per second, imagine how long it would take the bots to go through 10,000 pages. In order not to crash the server, spiders request fewer pages; this puts a slow site at a disadvantage to sites with faster load times. Chances are bots will index sites on a fast server more frequently and thoroughly than sites on a slow server. Page speed has become very important to Google in particular and so deserves some attention. We discuss improving page and site speed in depth in Book VII, Chapter 1.
The second variable is somewhat contested. Some SEOs believe that your rank could be affected by something called bounce rate, which measures how often someone clicks on a page and immediately hits the Back button. The search engines can detect when a user clicks on a result and then clicks on another result in a short time. If a website constantly has people loading the first page for only a few seconds before hitting the Back button to return to the search results, it’s a good bet that the website is probably not very satisfying. Remember, engines strive for relevancy and user experience in their results, so they most likely consider bounce rate when they're determining rankings.
So if all search engines are looking at these things, does it matter if you’re looking at Bing versus Google versus Yahoo? Yes, it does, because all search engines evaluate subject relevance differently. The big players have their own algorithms that measure things in a different way than their competition. So something that Google thinks belongs on Page 1 of listings may or may not pop up in the Top Ten over on Bing.
Getting to Know the Major Engines
It’s time to meet the three major search engines: Google, Bing, and Yahoo. As we said earlier, they all measure relevancy a bit differently. Google might rank a page of content as more relevant than Bing does, so Google’s results pages could look quite different from Bing’s results pages for the same search query. Meanwhile, Yahoo uses Bing’s index of the web and ranking algorithm to serve organic and much of its paid search listings. For this reason, deciding which search engine is best is often subjective. It all depends on whether you find what you’re looking for.
Organic versus paid results
One of the major ways search engines are differentiated is how they handle their organic versus paid results. Organic results are the web pages that the search engines find on their own using their spiders. Paid results (also called sponsored listings) are the listings that the site owners have paid for. In web searches, paid results usually appear as ads along the top or right side of the window, but they also can appear lower on the page, among or below organic listings. Paid results don’t necessarily match your search query either. Here’s how this happens.
Companies can bid on almost any keyword for which they want to get traffic (with some legal exceptions). The bid price needed to have an ad show in the SERP is based on many factors, including competition for the keyword, traffic on the keyword, and, at least in Google's case, the quality of the landing page. The better-constructed the landing page (the web page that a visitor receives after clicking an ad), the lower the minimum bid price. This doesn't have to be an exact match. Businesses often bid on keywords that are related to their products in hopes of catching more visitors. For example, if a visitor searches for tickets to Sports Team A, a sponsored (paid) link might show up advertising Sports Team B. This is what's happening below in Figure 2-1. Ticketmaster has bid on [Lakers tickets] as a keyword in order to advertise tickets for a different team, the LA Clippers, and clicking the sponsored link takes you to a page for buying Clippers tickets. The ad for Boston Celtics Tickets operates the same way. The organic links, however, should all take you to sites related to Lakers tickets.
Figure 2-1: The web search results for [Lakers tickets] include ads for two other sports teams’ tickets (outlined above).
Paid results are quite different from organic results. Generally, people click on organic results rather than paid results. You can't buy your way to the top of organic results. You can only earn your way there through effective search engine optimization.
Table 2-1 lists the major search engine players (in order of their appearance) and the attributes of each, for comparison. The following sections introduce you to each search engine in more detail and talk about its organic results and paid advertising services.
Table 2-1 U.S. Search Engine Comparison Table
Engine Name |
Organic |
Paid (Desktop) |
Paid (Mobile) |
Yahoo |
Yes. Uses Bing's index and algorithm |
Yes. Uses Bing Ads |
Yes. Yahoo Gemini |
Spider name: Googlebot |
Yes |
Yes. Google AdWords |
Yes. Google AdWords |
Bing Spider name: Bingbot |
Yes |
Yes. Bing Ads |
Yes. Bing Ads |
Yahoo
In 1994, two electrical engineering graduate students at Stanford University, David Filo and Jerry Yang, created Yahoo, a search engine and network of properties that has undergone much change in its 20-year history. For many years, Yahoo has outsourced its search function to other providers like Google and, today, Bing.
Organic results
In 2010, Yahoo made a deal with Bing to power its search engine and pay per click program. That means that the organic search results in Yahoo use Bing’s ranking algorithm and Bing’s index of the web.
Paid results
Because Yahoo no longer provides its own paid results for desktop searches, in order to advertise on Yahoo's network, you must use Microsoft Bing Ads (and get two-for-one exposure).
Mobile advertising is a different story. In 2014, Yahoo migrated to Yahoo Gemini, its own mobile ad marketplace. The strategy appears to be paying off; its third-quarter earnings place Yahoo in third place for total mobile ad revenue, behind Google and Facebook but, surprisingly, ahead of Twitter.
Google began as a research project by two other Stanford University students, Larry Page and Sergey Brin, in January 1996. They officially incorporated as Google in September 1998.
Organic results
Over time, Google has developed into the powerhouse of the search engine realm. Here are just some of the reasons why Google is the king of search engines and shows no signs of giving up the crown:
- Highly relevant: Google's relevancy is one of its strongest suits thanks to its analysis of site backlinks and on-site content on the one hand, and its understanding of user intent on the other.
- PageRank: PageRank is a famous part of Google’s search algorithm. PR is a numerical weight assigned to a website in order to measure its importance, based on backlinks.
- Enormous index: Google has indexed billions of pages on the Internet. In addition, Google search’s semantic-based algorithm and integration with other Google products (such as Gmail, Google+, YouTube, and Google Drive) give it a wide vantage point for seeing the connections between concepts, people, and other entities on the web.
- Advanced technology: Google is the undisputed leader, technologically speaking. Its algorithms aggressively weed out spam from its results, and its Knowledge Graph far outpaces the other engines’ capability to understand a searcher’s intent. You learn more about the Knowledge Graph and how it affects search results in Chapter 3 of this minibook.
- Brand recognition: Google is so recognized for search that the brand name is used as a verb and listed in dictionaries (as in, “I Googled this the other day …”).
- Most-visited web property: Google is the number one search engine worldwide (except in China, where Baidu reigns, and Russia, where Yandex tops the list). In the United States, Google has more of the search market than all the other search engines combined, netting more than two-thirds of all search engine traffic (see Table 2-2;
http://www.comscore.com/Insights/Press_Releases/2014/3/comScore_Releases_February_2014_U.S._Search_Engine_).
Table 2-2 comScore Explicit Core Search Share Report, U.S. (August versus July 2014)*
Core Search Entity |
July 2014 |
August 2014 |
Share Change |
Total Explicit Core Search |
100.0% |
100.0% |
None |
Google Sites |
67.4% |
67.3% |
–0.1 |
Microsoft Sites |
19.3% |
19.4% |
0.1 |
Yahoo Sites |
10.0% |
10.0% |
0.0 |
Ask Network |
2.0% |
2.0% |
0.0 |
AOL, Inc. |
1.3% |
1.3% |
0.0 |
*“Explicit Core Search” excludes contextually driven searches that do not reflect specific user intent to interact with the search results.
Paid results
Google has a service called Google AdWords that regulates its paid results for desktop and mobile. It’s a pay per click advertising model that lets you create your own ads, choose the keyword phrases you want your ad to appear for, and set your bid price and budget. Google ranks its ads based on the ad’s bid amount and Quality Score — a combined measure of the ad’s relevance, landing-page experience, and expected click-through rates, or how often the ad is clicked. Google AdWords can also help you create your ads if you’re stuck on how to do so.
After your AdWords ads are set up, Google matches your ads to the right audience within its network, and you pay only when your ad is clicked. Google provides gives an ever-increasing number of ways to target your ad audience, such as by demographics like gender, age group, annual household income, ethnicity, and number of children in the household. AdWords also allows you to do location-based targeting and day-parting, which limits the display of your advertisement to certain times of the day. Recently, Google introduced “In-market audiences” targeting, which lets you leverage Google’s understanding of consumer behavior patterns and find people who are actively researching products like yours.
You can potentially get a lot of exposure for your paid ads. The Google AdWords distribution network lets you advertise on Google search sites and affiliates like AOL and Ask.com. Mobile and tablet ads are also centrally managed through AdWords.
Google also offers the ability to publish ads on the Google Display Network. Consisting of more than two million websites, videos, and mobile apps, the Display Network also includes Google sites such as Gmail, YouTube, and Blogger. Owners can earn money by enrolling in Google’s AdSense program to allow advertisements on their sites, apps, or video content. AdSense ads generate revenue for the site owners based on factors such as clicks or impressions.
Bing
Bing (previously named MSN Search and Microsoft Live Search) is a search engine designed by Microsoft that competes with Google and Yahoo. It’s currently the second-most-used general search engine in the United States.
Organic results
In addition to providing rich, blended SERP results on par with Google’s, Bing differentiates itself through features like its daily full-screen home page image, a longer list of search results per page, and a rewards program that lets people earn points that can be redeemed for gift cards just for using Bing. Bing Search also lets users easily modify the search results based upon any location they would like the search to appear from.
Paid results
Microsoft’s paid program is called Bing Ads, and reports are that it offers advertisers an extremely good return on investment (ROI). Like Google, Bing ranks its ads based on keyword bid price and ad quality. Microsoft also lets you place adjustable bids based on demographic details. For example, a mortgage lead from an adult with a higher income might be worth more than an equivalent search by someone who is young and still in college.
Checking Out the Rest of the Field:AOL and Ask.com
The five biggest search engines worldwide right now are Google, Yahoo, Bing, Baidu (a Chinese search engine), and Yandex (a Russian search engine — see Book IX, Chapter 2 for more information on Baidu and Yandex), with Google taking home the lion’s share. But other smaller engines that draw a pretty respectable number of hits are still operating.
AOL
AOL has been around in some form or another since 1983. Today AOL provides some services such as email, chat, and its own search engine. AOL gets all its search engine results from Google, both organic and paid.
If you want to appear in an AOL search, you must focus on Google. AOL uses the Google index results in its search engine.
Ask.com
Ask.com (some of you may remember it as Ask Jeeves) pioneered blended search (the integration of different content types, such as images, videos, news, blogs, books, maps, and so on, onto the search results page) but failed to gain any significant market share from the larger three engines (Google, Bing, and Yahoo). Ask.com changed its market strategy and now considers itself an “answer engine,” rather than a search engine. Ask.com generates comprehensive results answering question-focused search queries, pulling information from third-party directories, feeds, Q&A forums, and Google’s organic results.
Ask.com gets most of its paid search ads from Google AdWords. Ask.com does have its own internal ad service, but it places internal ads above the Google AdWords ads only if it feels the internal ads will bring in more revenue.
Finding Your Niche: Vertical Engines
We’ve been talking mostly about general search engines, whose specific purpose is to scour everyone and everything on the web and return results to you. But there’s also another type of search engine known as a vertical search engine. Vertical search engines are search engines that restrict their search either by industry, geographic area, or file type. Google has several vertical search engines listed in the upper-left corner of its home page for images, apps, and so forth. So when you type [jam] into Google’s Images search, it only returns pictures of jam instead of web pages. The three main types of vertical search engines are detailed in the following sections.
Industry-specific
Industry-specific vertical search engines serve particular types of businesses. The real estate industry has its own search engines like Realtor.com (www.realtor.com) and Zillow.com (www.zillow.com), which provide housing listings, and companion sites like HomeAdvisor.com, which is for home improvement contractors. If you want to conduct searches related to the medical industry, you can use WebMD.com (www.webmd.com), a search engine devoted entirely to medical questions and services. Questions about Hollywood stars and anyone ever connected with movie-making can be answered at IMDB.com, which stands for Internet Movie Database. If you are searching for legal services, FindLaw.com (www.findlaw.com) and Lawyers.com (www.lawyers.com) can help you search for an attorney by location and practice.
Niche engines like these deliver less traffic but make up for it in the targeted nature of traffic. Visitors who access your site using niche engines are prequalified because they're looking for exactly your type of site.
Local
A local search engine is an engine specializing in websites that are tied to a limited physical area also known as a geotargeted area. Basically, this type of engine is looking for things in your general neck of the woods. In addition to their main indexes, the major search engines have local-only indexes that they integrate into their main results, like Google My Business and Bing Places. Businesses submit a local listing to the search engines, which includes information about the local business such as its name, address, city, phone number, categories the business may be searched under, and so on. That means the site could pop up if someone’s looking for that type of business within its geographic area.
A searcher might specify a ZIP code, city, or other geographic qualifier in a search query to find geotargeted results, such as [Milwaukee chiropractor]. However, Google and Bing can pretty reliably understand the intent of a search. If the engine thinks the searcher wants to find a nearby business, it displays local listings right in the search results. For example, a search for [dry cleaners] brings up website links to businesses near the searcher, even though no location was specified in the query.
Local results take up a lot of SERP space and often rank above the organic results. To be clear, these local results are still served up “organically” by the local search algorithm, but the local results may take precedence over what we have come to know in the past as the top organic results. Searches performed on a smartphone or other mobile device tend to have even more local results, since the majority of queries people speak or type into mobile devices are thought to have local intent. See Chapter 4 of this minibook for examples of several types of local results.
Many large cities have their own local search engines. Yahoo Local and Local.com are the most well known local-only engines. Internet directories like YellowPages.com, SuperPages.com, DexKnows.com, and YellowBook.com are also out there clamoring for your local search queries.
File type-specific
Certain search engines devote themselves to just one type of file — videos, photographs, or other. Rather than limiting traffic, being a file type-specific engine creates traffic because visitors know exactly what they’re going to get. The video-only engine YouTube.com, for example, reports some astonishing statistics: More than 1 billion people visit YouTube each month, watching more than 6 billion hours of video and uploading an average of 100 hours of video per minute. In fact, YouTube’s traffic makes it the second most popular website in the world, behind only Google (http://www.youtube.com/yt/press/statistics.html). There are other video engines as well, including Blinkx.com, Metacafe.com, and Hulu.com; Hulu specializes in the video niche of television shows and movies.
Image searches performed on Google, Bing, or Yahoo return images from all over the web. But there are also many successful image-only search engines, like PicSearch.com and CC (Creative Commons) Search (search.creativecommons.org), which comb the Internet looking for images that match your criteria. Sites like Flickr.com are image-sharing sites, where users can upload their photos and specify how they may be used. Other sites have their own stock of images that users can search for and download (either for free or for a fee based on the user’s intended use), such as GettyImages.com, iStockphoto.com, Fotolia.com, and Dreamstime.com, just to name a few.
As a website owner, one way to leverage these specialty engines is to upload content that will attract searchers looking for images or videos. You can learn more about strategies with images and videos in Book V, Chapter 2.
Discovering Internal Site Search
Say you’re writing an article and you need to reference something in the New York Times. The New York Times website (www.nytimes.com) keeps an archive of online articles, but because you can’t remember the date the article was published, you’d have a long trek through the online archives. Luckily, it has its own internal site search engine that enables you to look up articles using keywords and filters. Any search engine that’s site-specific, meaning it searches just that website, is an internal site search engine.
Larger websites with thousands of pages employ these internal site search engines as an easy way of browsing their archives. A very small site probably doesn't need an internal search, but most e-commerce sites with more than a few products should consider implementing one.
Techniques that help you rank in general search engines also help your users when they need to find something on your site using an internal search. A good internal search can be the difference between making a sale and visitors leaving in frustration. To get started quickly, Google offers a hosted internal search solution as well as an enterprise-level solution. See https://www.google.com/enterprise/search/products/gss.html for more information.
Understanding Metasearch Engines
Another breed of search engine to be aware of is a metasearch engine. Metasearch engines do not maintain a database of their own, but instead combine results from multiple search engines. The advantage they tout is a twist on “bigger is better” — the more results you see in one fell swoop, the better. The sites Dogpile.com (www.dogpile.com) and DuckDuckGo.com (www.duckduckgo.com) top the list of metasearch engines. When you run a search on Dogpile, it pulls and displays results from four of the largest global engines (including Google and Yahoo) in one place. DuckDuckGo’s marketing proposition is that it respects searchers’ privacy by not tracking their search behavior, but pulls from many different search engines and key sites to present non-personalized search results.
After pulling results from multiple search engines, the metasearch engines retain the top ranked results from the separate search engines and present the user with the top results. This is different from applying an algorithm, as the indexed search engines do (an algorithm is a mathematical equation that weighs many specific criteria about each web page to generate its ranking result, as we discuss in Chapter 1 of this minibook). Metasearch engines take more of a filtering approach to all the indexed data gathered from the other search engines. Ads appear at the top and bottom, with organic “Web Results” in the middle of the page. We’ve found metasearch engines helpful for keeping track of which competitors buy paid results for which keywords. (You can read more about paid results in Book I, Chapter 4). Figure 2-2 shows you a results page from the Dogpile metasearch engine.
Figure 2-2: The Dogpile metasearch engine results page.