A synonym ring (see Figure 9-2) connects a set of words that are defined as equivalent for the purposes of retrieval. In practice, these words are often not true synonyms. For example, imagine you’re redesigning a consumer portal that provides ratings information about household products from several companies.

Figure 9-2. A synonym ring

When you examine the search logs and talk with users, you’re likely to find that different people looking for the same thing are entering different terms. Someone who’s buying a food processor may enter “blender” or one of several product names (or their common misspellings). Take a look at the content, and you’re likely to find many of these same variations.

There may be no preferred terms, or at least no good reason to define them. Instead, you can use the out-of-the-box capabilities of a search engine to build synonym rings. This can be as simple as entering sets of equivalent words into a text file. When a user enters a word into the search engine, that word is checked against the text file. If the word is found, then the query is “exploded” to include all of the equivalent words. For example, in Boolean logic:

(kitchenaid) becomes (kitchenaid or "kitchen aid" or blender or 
"food processor" or cuisinart or cuizinart)

What happens when you don’t use synonym rings? Consider Figure 9-3, which shows the results of a search for “pocketpc.” Pretty discouraging, huh? Looks like we might have to look elsewhere. But look what happens when we put a space between “pocket” and “pc” (Figure 9-4).

Figure 9-3. Results of a search at Computershopper

Figure 9-4. Another search on the same site

Suddenly, the site has oodles of information about the Pocket PC. A simple synonym ring linking “pocketpc” and “pocket pc” would solve what is a common and serious problem from both user and business perspectives.

However, synonym rings can also introduce new problems. If the query term expansion operates behind the scenes, users can be confused by results that don’t actually include their keywords. In addition, the use of synonym rings may result in less relevant results. This brings us back to the subject of precision and recall.

As you may recall from Chapter 8, precision refers to the relevance of documents within a given result set. To request high precision, you might say, “Show me only the relevant documents.” Recall refers to the proportion of relevant documents in the result set compared to all the relevant documents in the system. To request high recall, you might say, “Show me all the relevant documents.” Figure 9-5 shows the mathematics behind precision and recall ratios.

Figure 9-5. Precision and recall ratios

While both high precision and high recall may be ideal, it’s generally understood in the information retrieval field that you usually increase one at the expense of the other. This has important implications for the use of controlled vocabularies.

As you might guess, synonym rings can dramatically improve recall. In one study conducted at Bellcore in the 1980s,^[1] the use of synonym rings (they called it “unlimited aliasing”) within a small test database increased recall from 20 to 80 percent. However, synonym rings can also reduce precision. Good interface design and an understanding of user goals can help strike the right balance. For example, you might use synonym rings by default but order the exact keyword matches at the top of the search results list. Or, you might ignore synonym rings for initial searches but provide the option to “expand your search to include related terms” if there were few or no results.

In summary, synonym rings are a simple, useful form of vocabulary control. There is really no excuse for the conspicuous absence of this basic capability on many of today’s largest web sites.

Strictly defined, an authority file is a list of preferred terms or acceptable values. It does not include variants or synonyms. Authority files have traditionally been used largely by libraries and government agencies to define the proper names for a set of entities within a limited domain.

As shown in Figure 9-6, the Utah State Archives & Records Service has published a listing of the authoritative names of public institutions in the state of Utah. This is primarily useful from content authoring and indexing perspectives. Authors and indexers can use this authority file as the source for their terms, ensuring accuracy and consistency.

Figure 9-6. An authority file

In practice, authority files are commonly inclusive of both preferred and variant terms. In other words, authority files are synonym rings in which one term has been defined as the preferred term or acceptable value.

The two-letter codes that constitute the standard abbreviations for U.S. states as defined by the U.S. Postal Service provide an instructive example. Using the purist definition, the authority file includes only the acceptable codes:

AL, AK, AZ, AR, CA, CO, CT, DE, DC, FL, GA, HI, ID, 
IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, 
NJ, NM, NY, NC, ND, OH, OK, OR, PA, PR, RI, SC, SD, TN, TX, UT, VT, 
VA, WA, WV, WI, WY.

However, to make this list useful in most scenarios, it’s necessary to include, at a minimum, a mapping to the names of states:

AL Alabama
AK Alaska
AZ Arizona
AR Arkansas
CA California
CO Colorado
CT Connecticut
 . . . 

To make this list even more useful in an online context, it may be helpful to include common variants beyond the official state name:

CT Connecticut, Conn, Conneticut, Constitution State

At this point, we run into some important questions about the use and value of authority files in the online environment. Since users can perform keyword searches that map many terms onto one concept, do we really need to define preferred terms, or can synonym rings handle things just fine by themselves? Why take that extra step to distinguish CT as the acceptable value?

First, there are a couple of backend reasons. An authority file can be a useful tool for content authors and indexers, enabling them to use the approved terms efficiently and consistently. Also, from a controlled vocabulary management perspective, the preferred term can serve as the unique identifier for each collection of equivalent terms, allowing for more efficient addition, deletion, and modification of variant terms.

There are also a number of ways that the selection of preferred terms can benefit the user. Consider Figure 9-7, where Drugstore.com is providing a mapping between the equivalent term “tilenol” and the authoritative brand name, “Tylenol.” By showing users the preferred terms, you can educate them. In some cases, you’ll be helping them to correct a misspelling. In others, you may be explaining industry terminology or building brand recognition.

Figure 9-7. Mapping between equivalent terms

These “lessons” may be useful in very different contexts, perhaps during the next telephone conversation or in-store interaction a customer has with your organization. It’s an opportunity to nudge everyone toward speaking the same language, without assuming or requiring such conformity within the search system. In effect, the search experience can be similar to an interaction with a sales professional, who understands the language of the customer and translates it back to the customer using the company or industry terminology.

Preferred terms are also important as the user switches from searching to browsing mode. When designing taxonomies, navigation bars, and indexes, it would be messy and overwhelming to present all of the synonyms, abbreviations, acronyms, and common misspellings for every term.

At Drugstore.com, only the brand names are included in the index (see Figure 9-8); equivalent terms like “tilenol” don’t show up. This keeps the index relatively short and uncluttered, and in this example, reinforces the brand names. However, a trade-off is involved. In cases where the equivalent terms begin with different letters (e.g., aspirin and Bayer), there is value in creating pointers:

Aspirin see Bayer

Figure 9-8. Brand index at Drugstore.com

Otherwise, when users look in the index under A for aspirin, they won’t find Bayer. The use of pointers is called term rotation. Drugstore.com doesn’t do it at all. To see a good example of term rotation used in an index to guide users from variant to preferred terms, we’ll switch to the financial services industry.

In Figure 9-9, users looking for “before-tax contributions” are guided to the preferred term “pretax contributions.” Such integration of the entry vocabulary can dramatically enhance the usefulness of the site index. However, it needs to be done selectively; otherwise, the index can become too long, harming overall usability. Once again, a careful balancing act is involved that requires research and good judgment.

Figure 9-9. A site index with term rotation

We use classification scheme to mean a hierarchical arrangement of preferred terms. These days, many people prefer to use taxonomy instead. Either way, it’s important to recognize that these hierarchies can take different shapes and serve multiple purposes, including:

Consider, for example, the Dewey Decimal Classification (DDC). First published in 1876, the DDC is now “the most widely used classification scheme in the world. Libraries in more than 135 countries use the DDC to organize and provide access to their collections.”^[2] In its purest form, the DDC is a hierarchical listing that begins with 10 top-level categories and drills down into great detail within each.

000 Computers, information, & general reference
100 Philosophy & psychology
200 Religion
300 Social sciences
400 Language
500 Science
600 Technology
700 Arts & recreation
800 Literature
900 History & geography

For better or worse, the DDC finds its way into all sorts of interface displays. As Figure 9-10 shows, the National Library of Canada uses it as a browsable hierarchy.

Figure 9-10. The Dewey Decimal Classification in action

Classification schemes can also be used in the context of searching. Yahoo! does this very effectively. You can see in Figure 9-11 that Yahoo!’s search results present “Category Matches,” which reinforces users’ familiarity with Yahoo!’s classification scheme.

Figure 9-11. Category Matches at Yahoo!

The important point here is that classification schemes are not tied to a single view or instance. They can be used on both the back end and the front end in all sorts of ways. We’ll explore types of classification schemes in more detail later in this chapter, but first let’s take a look at the “Swiss Army Knife” of vocabulary control, the thesaurus.

Dictionary.com defines thesaurus as a “book of synonyms, often including related and contrasting words and antonyms.” This usage hearkens back to our high school English classes, when we chose big words from the thesaurus to impress our teachers.

Our species of thesaurus, the one integrated within a web site or intranet to improve navigation and retrieval, shares a common heritage with the familiar reference text but has a different form and function. Like the reference book, our thesaurus is a semantic network of concepts, connecting words to their synonyms, homonyms, antonyms, broader and narrower terms, and related terms.

However, our thesaurus takes the form of an online database, tightly integrated with the user interface of a web site or intranet. And though the traditional thesaurus helps people go from one word to many words, our thesaurus does the opposite. Its most important goal is synonym management—the mapping of many synonyms or word variants onto one preferred term or concept—so the ambiguities of language don’t prevent people from finding what they need.

So, for the purposes of this book, a thesaurus is:

A controlled vocabulary in which equivalence, hierarchical, and associative relationships are identified for purposes of improved retrieval.^[3]

A thesaurus builds upon the constructs of the simpler controlled vocabularies, modeling these three fundamental types of semantic relationships.

As you can see from Figure 9-12, each preferred term becomes the center of its own semantic network. The equivalence relationship is focused on synonym management. The hierarchical relationship enables the classification of preferred terms into categories and subcategories. The associative relationship provides for meaningful connections that aren’t handled by the hierarchical or equivalence relationships. All three relationships can be useful in different ways for the purposes of information retrieval and navigation.

Figure 9-12. Semantic relationships in a thesaurus

A classic thesaurus is used at the point of indexing and at the point of searching. Indexers use the thesaurus to map variant terms to preferred terms when performing document-level indexing. Searchers use the thesaurus for retrieval, whether or not they’re aware of the role it plays in their search experience. Query terms are matched against the rich vocabulary of the thesaurus, enabling synonym management, hierarchical browsing, and associative linking. This is the full-bodied, fully integrated thesaurus we’ve referred to for much of this chapter.

However, building a classic thesaurus is not always necessary or possible. Consider a scenario in which you have the ability to develop a controlled vocabulary and index documents, but you’re not able to build the synonym-management capability into the search experience. Perhaps another department owns the search engine and won’t work with you, or perhaps the engine won’t support this functionality without major customization.

Whatever the case, you’re able to perform controlled vocabulary indexing, but you’re not able to leverage that work at the point of searching and map users’ variant terms to preferred terms. This is a serious weakness, but there are a few reasons why an indexing thesaurus may be better than nothing:

Such consistency of indexing can provide real value for information systems with captive audiences. When dealing with an intranet application that’s used by the same people on a regular basis, you can expect these users to learn the preferred terms over time. In such an environment, indexing consistency begins to rival indexing quality in value.

And finally, an indexing thesaurus positions you nicely to take the next step up to a classic thesaurus. With a vocabulary developed and applied to your collection of documents, you can focus your energies on integration at the user interface level. This may begin with the addition of an entry vocabulary to your browsable indexes and will hopefully bring searching into the fold, so the full value of the thesaurus is used to power the searching and browsing experience.

Sometimes a classic thesaurus isn’t practical because of issues on the content side of the equation that prevent document-level indexing. Perhaps you’re dealing with third-party content or dynamic news that’s changing every day. Perhaps you’re simply faced with so much content that manual indexing costs would be astronomical. (In this case, you may be able to go with a classic thesaurus approach that leverages automated-categorization software, as described in Chapter 16.) Whatever the case, there are many web and intranet environments in which controlled vocabulary indexing of the full document collection just isn’t going to happen. This doesn’t mean that a thesaurus isn’t still a viable option to improve the user experience.

A searching thesaurus leverages a controlled vocabulary at the point of searching but not at the point of indexing. For example, when a user enters a term into the search engine, a searching thesaurus can map that term onto the controlled vocabulary before executing the query against the full-text index. The thesaurus may simply perform equivalence term explosion, as we’ve seen in the case of synonym rings, or it may go beyond the equivalence relationship, exploding down the hierarchy to include all narrower terms (traditionally known as “posting down”). These methods will obviously enhance recall at the expense of precision.

You also have the option of giving more power and control to the users—asking them whether they’d like to use any combination of preferred, variant, broader, narrower, or associative terms in their query. When integrated carefully into the search interface and search result screens, this can effectively arm users with the ability to narrow, broaden, and adjust their searches as needed.

A searching thesaurus can also provide greater browsing flexibility. You can allow your users to browse part or all of your thesaurus, navigating the equivalence, hierarchical, and associative relationships. Terms (or the combination of preferred and variant terms) can be used as predefined or “canned” queries to be run against the full-text index. In other words, your thesaurus can become a true portal, providing a new way to navigate and gain access to a potentially enormous volume of content. A major advantage of the searching thesaurus is that its development and maintenance costs are essentially independent of the volume of content. On the other hand, it does put much greater demands on the quality of equivalence and mapping.

If you’d like to learn more about searching thesauri, try these articles:

The equivalence relationship (Figure 9-21) is employed to connect preferred terms and their variants. While we may loosely refer to this as “synonym management,” it’s important to recognize that equivalence is a broader term than synonymy.

Figure 9-21. The equivalence relationship

Our goal is to group terms defined as “equivalent for the purposes of retrieval.” This may include synonyms, near-synonyms, acronyms, abbreviations, lexical variants, and common misspellings; for example:

In the case of a product database, it may also include the names of retired products and of competitors’ products. Depending on the desired specificity of your controlled vocabulary, you may also fold more general and more specific terms into the equivalence relationship to avoid extra levels of hierarchy. The goal is to create a rich entry vocabulary that serves as a funnel, connecting users with the products, services, and content that they’re looking for and that you want them to find.

The hierarchical relationship (Figure 9-22) divides up the information space into categories and subcategories, relating broader and narrower concepts through the familiar parent-child relationship.

Figure 9-22. The hierarchical relationship

There are three subtypes of hierarchical relationship:

At first blush, the hierarchical relationship sounds pretty straightforward. However, anyone who’s ever developed a hierarchy knows that it isn’t as easy as it sounds. There are many different ways to hierarchically organize any given information space (e.g., by subject, by product category, or by geography). As we’ll explain shortly, a faceted thesaurus supports the common need for multiple hierarchies. You also need to deal with the tricky issues of granularity, defining how many layers of hierarchy to develop.

Once again, we need to ground our work in the ultimate goal of enhancing the ability of our users to find what they need. The card-sorting methodologies (discussed in Chapter 10) can help you begin to shape your hierarchies based on user needs and behaviors.

The associative relationship (Figure 9-23) is often the trickiest, and by necessity is usually developed after you’ve made a good start on the other two relationship types. In thesaurus construction, associative relationships are often defined as strongly implied semantic connections that aren’t captured within the equivalence or hierarchical relationships.

Figure 9-23. The associative relationship

There is the notion that associative relationships should be “strongly implied.” For example, hammer RT nail. In practice, however, defining these relationships is a highly subjective process.

The ANSI/NISO thesaurus discusses many associative relationship subtypes. For example:

In the world of e-commerce, the associative relationship provides an excellent vehicle for connecting customers to related products and services. Associative relationships allow what marketing folks call “cross-selling,” allowing an e-commerce site, for example, to say “Hey, nice pants! They’d go great with this shirt.” When done well, these associative relationships can both enhance the user experience and further the goals of the business.

Defining the form of preferred terms is something that seems easy until you try it. All of a sudden, you find yourself plunged into heated arguments over grammatical minutiae. Should we use a noun or a verb? What’s the “correct” spelling? Do we use the singular or plural form? Can an abbreviation be a preferred term? These debates can suck up large amounts of time and energy.

Fortunately, the ANSI/NISO thesaurus standard goes into great detail in this area. We recommend following these guidelines, while allowing for exceptions when there’s a clear benefit. Some of the issues covered by the standard include:

Of course, selection of a preferred term involves more than the form of the term; you’ve got to pick the right term in the first place. The ANSI/NISO standard won’t help too much here. Consider the following excerpts:

Section 3.0. “Literary warrant (occurrence of terms in documents) is the guiding principle for selection of the preferred (term).”

Section 5.2.2. “Preferred terms should be selected to serve the needs of the majority of users.”

This tension between literary warrant and user warrant can be resolved only by reviewing your goals and considering how the thesaurus will be integrated with the web site. Do you want to use preferred terms to educate your users about the industry vocabulary? Will you be relying on preferred terms as your entry vocabulary (e.g., no variants in the index)? You’ll need to answer these questions before deciding on the primary source of authority for term selection.

Within the thesaurus itself, we’re striving for extreme specificity in our use of language. Remember, we’re trying to control vocabulary. Beyond the selection of distinctive preferred terms, there are some tools for managing ambiguity.

Parenthetical term qualifiers provide a way to manage homographs. Depending on the context of your thesaurus, you may need to qualify the term “Cells” in some of the following ways:

Scope notes provide another way to increase specificity. While they can sometimes look very much like definitions, scope notes are a different beast. They are intended to deliberately restrict meaning to one concept, whereas definitions often suggest multiple meanings. Scope notes are very useful in helping indexers to select the right preferred term. They can sometimes be leveraged in searching or results display to assist users as well.

The specificity of terms is another difficult issue that all thesaurus designers must face. For example, should “knowledge management software” be represented as one term, two terms, or three terms? Here’s what the standards have to say:

ANSI/NISO Z39.19. “Each descriptor . . . should represent a single concept.”

ISO 2788. “It is a general rule that . . . compound terms should be factored (split) into simple elements.”

Once again, the standards don’t make your life easy. ANSI/NISO leaves you arguing over what constitutes a “single concept.” ISO leads you toward uniterms (e.g., knowledge, management, software), which would probably be the wrong way to go in this example.

You need to strike a balance based on your context. Of particular importance is the size of the site. As the volume of content grows, it becomes increasingly necessary to use compound terms to increase precision. Otherwise, users get hundreds or thousands of hits for every search (and every preferred term).

The scope of content is also important. For example, if we’re working on a web site for Knowledge Management magazine, the single term “knowledge management software” or perhaps “software (knowledge management)” may be the way to go. However, if we’re working on a broad IT site like CNET, it may be better to use “knowledge management” and “software” as independent preferred terms.