You know your application model, right? You can probably list the five most important objects represented in your system without even breaking a sweat. In fact, a good place to start is by composing an “elevator pitch” for your application; in other words, describe, as succinctly as you can, what your application will do. Here are some examples:

These are short, sweet, and somewhat vague descriptions of three very different applications, but each one still tells us much of what we need to know to plan our reputation needs. The recipe sharing site likely will benefit from some form of reputation for recipes and will require some way for users to rate them. The shopping lists? Not so much; those are more like utilities for individual users to manage the application data.

As for the music site, perhaps it’s not any one track that’s worth assessing but the quality of the playlists. Or maybe what’s relevant is users’ reputations as deejays (their performance, over time, at building and sustaining an audience). Tracking either one will probably also require keeping mashup tracks as an object in the system, but those tracks may not necessarily be treated as primary reputable entities.

In the intranet application for paralegals, the briefs are the primary atomic unit of interest. Who saved what briefs, how are users adding metadata, and how many people attach themselves to a document? These are all useful bits of information that will help filter and rank briefs to present back to other users.

So the first step toward defining the relevant objects in your reputation system is to start with what’s important in the application model, then think forward a little to what types of problems reputation will help your users solve. Then you can start to catalog the elements of your application in a more formal fashion.

Although you’ve thought at a high level about the primary objects in your application model, you’ve probably overlooked some smaller-order, secondary objects and concepts. These primary and secondary objects relate to one another in interesting ways that we can make use of in our reputation system. An application audit can help you to fully understand the entities and relationships in your application.

Make a complete inventory of every kind of object in your application model that may have anything to do with accomplishing your goals. Some obvious items are user profile records and data objects that are special to your application model: movies, transactions, landmarks, CDs, cameras, or whatever. All of these are clear candidates for tracking as reputable entities.

It is very important to know what objects will be the targets of reputation statements and any new objects you’ll need to create for that purpose. Make sure you understand the metadata surrounding each object and how your application will access it. How are the objects organized? Are they searchable by attributes? Which attributes? How are the different objects related to one another?

Some objects in your application model will be visually represented in the interface, so one way to start an audit is with a simple survey of screen designs, at whatever fidelity is available. For in-progress projects, early-stage wireframes are fine—if your application is in production, take some screenshots and print them. Figure 6-1 shows an audit-screengrab for a project already in production, Yahoo! Message Boards.

Figure 6-1. Here, users, threads, and posts are all useful entities to model in a message board context.

Also be sure to list items whose creation, editing, or content can be used as input into your reputation model. Some common types of such items are:

Spend some time considering more nuanced sources of information, such as historical activity data, external reputation information, and special processes that provide application-relevant insight.

As an example, consider a profanity filter applied to user-supplied text messages, replacing positive matches with asterisks (****); the real-time results of applying the filter might provide a way to measure the effects of interface changes on this particular user behavior (the use of profanity).

This probably should go without saying, but—what the heck—let’s say it anyway. If your entire application offering is built around a specific type of object, social or otherwise, the object is probably a good candidate for tracking as a reputable entity.

And remember, nothing interests people more than…other people. That phenomenon alone is an argument for at least considering using karma (people reputation) in any application that you might build. Users will always want every possible edge in understanding other actors in a community. What motivates them? What actions have they performed in the past? How might they behave in the future?

When you’re considering what objects will be of the most interest to your users, don’t overlook other, related objects that also may benefit users. For example, on a photo-sharing site, it seems natural that you’d track a photo’s reputation. But what about photo albums? They may not be the very first application object that you think of, but in some situations, it’s likely you’ll want to direct users’ attention to high-quality groupings or collections of photos.

The solution is to track both objects. Each may affect the reputation of the other to some degree, but the inputs and weightings you’ll use to generate reputation for each will necessarily differ.

Some decisions are easy to make, requiring little investment of time or effort. Likewise, the cost of recovering from such decisions is negligible. For example, suppose I ask myself, “Should I read this blog entry?”

If it’s a short entry, and I’m already engaged in the act of reading blogs, and no other distractions are calling me away from that activity, then, yeah, I’ll probably go ahead and give it a read. (Of course, the content of the entry itself is a big factor. If neither the entry’s title nor a quick skim of the body interests me, I’ll likely pass it up.) And, once I’ve read it, if it turns out not to have been a very good choice? Well, no harm done. I can recover gracefully, move on to the next entry in my feed reader, and proceed on my way.

The level of decision investment is important because it affects the likelihood that a user will make use of available reputation information for an item. In general, the greater the investment in a decision, the more a user will expect (and make use of) robust supporting data (see Figure 6-2).

Figure 6-2. More critical decisions demand more complete information.

So for high-investment decisions (such as purchasing or any other decision that is not easily rescinded), offer more robust mechanisms, such as reputation information, for use in decision making.

You can do a lot with design to “ease” the presence of reputation inputs in your application interface. Sites continue to get more and more sophisticated with incorporating explicit and implicit controls for stating an entity value. Furthermore, the web-using public is becoming more and more accepting of, and familiar with, popular voting and favoriting input mechanisms.

Neither of these facts, however, obviates this requirement: reputable entities themselves must have some intrinsic value apart from the reputation system. Ask your users to participate only in ways that are appropriate in relation to that object’s intrinsic value. Don’t ask users for contributions (such as reviews or other metadata) that add value to an object whose intrinsic apparent value is low.

It might be OK, for example, to ask someone to give a thumbs-up rating to someone else’s blog comment (because the cost to the user providing the rating is low—basically a click). But it would be inappropriate to ask for a full-blown review of the comment. Writing the review would require more effort and thought than there was in the initial comment.

Reputable entities must remain in the community pool long enough for all members of the community to cast a vote (Figure 6-3). There’s little use in asking users for metadata for an item if other users cannot come along afterward and enjoy the benefit of that metadata.

Figure 6-3. The value of ratings increases with the time value of the content being rated.

Highly ephemeral items, such as news articles that disappear after 48 or 72 hours, probably aren’t good candidates for certain types of reputation inputs. For example, you wouldn’t ask users to author a multi-part review of a news story destined to vanish in less than a day, but you might ask them to click a “Digg this” or “Buzz this” button.

Items with a great deal of persistence (such as real-world establishments like restaurants or businesses) make excellent candidates for reputation. Furthermore, it can be appropriate to ask users for more involved types of inputs for persistent items, because it’s likely that other users will have a chance to benefit from the work that the community puts into contributing content.

Explicit claims represent your community’s voice and opinion. They operate through interface elements you provide that solicit users’ opinions about an entity, good or bad. A fundamental difference exists between explicit claims and implicit ones (discussed below), which boils down to user intent and comprehension.

With explicit claims, users should be fully aware that the action they’re performing is intended as an expression of an opinion. That intent differs greatly from the ones for implicit claims, in which users mostly just go about their business, generating valuable reputation information as a side effect.

Any time a user takes some action in relation to a reputation entity, it is very likely that you can derive valuable reputation information from that action. Recall the discussion of implicit and explicit reputation claims in The Reputation Statement. With implicit reputation claims, we watch not what the user says about the quality of an entity but how they interact with that object. For example, assume that a reputable entity in your system is a text article. You’ll find valuable reputation information in the answers to the following questions:

You can construct any number of smart, nuanced, and effective reputation models from relevant related action-type claims. Your only real limitation is the level and refinement of instrumentation in your application: are you prepared to capture relevant actions at the right junctures and generate reputation events to share with the system?

Don’t continue to reward people or objects for performing the same action over and over; rather, try to single out events that indicate that the target of the claim is worth paying attention to. For instance, the act of bookmarking an article is probably a more significant event than a number of page views. Why? Because bookmarking something is a deliberate act—the user has assessed the object in question and decided that it’s worth further action.

The process of creating karma is subtly more complex and socially delicate than creating reputation for things. For a deeper explanation, see Karma. Yahoo! has a general community policy of soliciting explicit ratings input only for user-created content—never having users directly rate other users. This is a good practice for a number of reasons:

It’s often worthwhile to reward firsts for users in your system, perhaps rewarding the first user to “touch” an object (leave a review, comment on a story, or post a link). Or, conversely, you might reward a whole host of firsts for a single user (to encourage him to interact with a wide range of features, for instance).

But once a first has been acknowledged, don’t continue to reward users for more of the same.

Pick events that are hard for users to replicate; this combats gaming of the system. But anticipate these patterns of behavior anyway, and build a way to deal with offenders into your system.

In Ratings bias effects, we discussed ratings distributions and why it’s important to pay attention to them. If you’re seeing data with poorly actionable distributions (basically, data that doesn’t tell you much), it’s likely that you’re asking for the wrong inputs.

Pay attention to the context in which you’re asking. For example, if interest in the object being rated is relatively low (perhaps it’s official feed content from a staid, corporate source), 5-star ratings are probably overkill. Your users won’t have such a wide range of opinions about the content that they’ll need five stars to judge it.

Ask for information in a way that’s consistent and appropriate with how you’re going to use it. For example, if your intent is to display the community average rating for a movie on a scale of 1 to 5 stars, it makes the most sense to ask users to enter movie ratings on a scale of 1 to 5 stars. Of course, you can transform reputation scores and present them back to the community in different ways (see the sidebar What Comes in Is Not What Goes Out), but strive to do that only when it makes sense and in a way that doesn’t confuse your users.

Before we dive into all the ways in which users might provide explicit feedback about objects in a system, think about the context in which users provide feedback. Remember, this book is organized around the reputation information that users generate—usable metadata about objects and about other people. But users themselves have a different perspective, and their focus is often on other matters altogether (see Figure 6-5). Feeding your reputation system is likely the last thing on their mind.

Figure 6-5. Even though this flow is idealized, it amply demonstrates that the “simple” act of rating something is actually pretty complex.

Given the other priorities, goals, and actions that your users might possibly be focused on at any given point during their interaction with your application, here are some good general guidelines for gathering explicit reputation inputs effectively.

A number of different input mechanisms let users express an opinion about an object across a range of values. A very typical such mechanism is star ratings. Yahoo! Local (see Figure 6-6) allows users to rate business establishments on a scale of 1 to 5 stars.

Figure 6-6. Yahoo! design standards specify that the mechanism for entering your own rating must always look different from the mechanism for displaying the community average rating.

Stars seem like a pretty straightforward mechanism, both for your users to consume (5-star rating systems seem to be everywhere, so users aren’t unfamiliar with them) and for you, the system designer, to plan. Tread carefully, though. Here are some small behavioral and interaction “gotchas” to think about early, during the design phase.

“Thumb” voting (thumb up or thumb down) lets a user quickly rate content in a fun, engaging way (see Figure 6-7). The benefit to the user for voting is primarily related to self-expression (“I love this!” or “I hate this!”). The ratings don’t need to be presented visually as thumbs (in fact, sometimes they shouldn’t), but in this book, we use “thumb” as shorthand for a two-state voting mechanism.

Figure 6-7. Yahoo! TV lets viewers rate their favorite episodes with thumb-up or thumb-down votes. The highly-skewed results shown in this example are typical: it turns out that only series fans go through the bother of rating individual episodes.

Thumb voting allows users to express strongly polarized opinions about assets. For example, if you can state your question as simply as “Did you like this or not?”, thumb voting may be appropriate. If it seems more natural to state your question as “How much did you like this?”, then star ratings seem more appropriate.

A popular and effective use of two-state voting is as a meta-moderation device for user-submitted opinions, comments, and reviews, as in Figure 6-8. Wherever you solicit user opinions about an object, also consider letting the community voice opinions about that opinion, by providing an easy control such as “Was this helpful?” or “Do you agree?”

Figure 6-8. Two-state voting works best when the interface presents a clear and concise user prompt. Leave no doubt about what you’re asking users to decide: “Was this review helpful to you?”

Avoid thumb voting for multiple facets of an entity. For example, don’t provide multiple thumb widgets for a product review intended to record users’ satisfaction with the product’s price, quality, design, and features. Generally, a thumb vote should be associated with an object in a one-to-one relationship: one entity gets one thumb up or down. Think of the metaphor, after all: Emperor Nero never would have let a gladiator’s arm survive while putting his leg to death. Think of thumb voting as an all-or-nothing rating.

Consider thumb voting when you want a fun, lightweight rating mechanism. The context for thumb voting should be appropriately fun and lighthearted, too; don’t use thumb voting in contexts where it will appear insensitive or inappropriate.

Don’t use thumb ratings when you want to compare qualitative data among assets. For example, in a long list of movies available for rental, you may want to permit sorting of the list by average rating. If you have been collecting thumb ratings, such sorting wouldn’t be very useful to users. (Instead, consider a 5-star scalar rating style.)

Vote to promote fulfills a very specific niche in the world of online opinion-gathering. As users browse a collection or pool of media objects, they mark items as worthwhile using a control consisting of a simple gesture. This pattern has been popularized by social news sites such as Digg, Yahoo! Buzz, and Newsvine.

Typically, these votes accumulate and are used to change the rank of items in the community pool and present winners with more prominence or a higher status, but that’s not an absolute necessity. Facebook offers an “I like this” link (Figure 6-9 ) that simply communicates a user’s good will toward an item.

Figure 6-9. Facebook’s “Like” vote is a tiny, personal vote-to-promote-style system. A large part of its value to you, the user, comes from your social connection to the voters.

The names of users who “like” an item are displayed to other users who encounter the item (until the list of vote casters becomes cumbersome, when the display switches to a sort of summary score), but highly liked items aren’t promoted above other items in the news feed in any obvious or overt way.

Writing a user review demands a lot of effort from users. It is among the most involved explicit input mechanisms that you can present, usually consisting of detailed, multi-part data entry (see Figure 6-10).

To get good, usable comparison data from user reviews, try to ensure that the objects you offer up for review meet all the criteria that we listed above for good reputable entities (see What Makes for a Good Reputable Entity?). Ask users to write reviews only for objects that are valuable, long-lived, and that have a high-investment decision.

Figure 6-10. Amazon’s review-writing process—while streamlined—still asks a lot of review writers. This barrier to participation may skew involvement from the community.

Reviews typically are compound reputation claims, with each review made up of a number of smaller inputs bundled together. You might consider any combination of the following for your user-generated reviews:

Any time a user saves an object or marks it for future consideration—either for himself or to pass along to someone else—that could be a reputable event.

This type of input has a lot in common with the explicit vote-to-promote input (see Vote to promote: Digging, liking, and endorsing), but the difference lies in user intent and perception: in a vote-to-promote system, the primary motivator for marking something as worthwhile is to publicly state a preference, and the expectation is that information will be shared with the community. It is a more extrinsic motivation.

By contrast, favorites, forwarding, and adding to a collection are more intrinsically motivated reputation inputs—actions that users take for largely private purposes.

Forwarding

You might know this input pattern as “send to a friend.” This pattern facilitates a largely private communication between two friends, in which one passes a reputable entity on to another for review. Yahoo! News has long promoted most emailed articles as a type of currency-of-interest proxy reputation. (See Figure 6-11.)

Figure 6-11. A danger with implicit inputs like send-to-a-friend is that, once the community becomes accustomed to their presence, they risk becoming a type of explicit statement. Do you suppose Yahoo! News has to battle email bots designed to artificially inflate articles’ visibility on the site?

Greater disclosure is a highly variable input: there’s a wide range of ways to present it in an interface (and weight it in reputation processes). But if users request “more information” about an object, you might consider those requests as a measure of the interest in that object—especially if users are making the requests after having already evaluated some small component of the object, such as an excerpt, a thumbnail, or a teaser.

A common format for blogs, for instance, is to present a menu of blog entries, with excerpts for each, and “read more” links. Clicks on a link to a post may be a reasonably accurate indicator of interest in the destination article.

But beware—the limitations of a user interface can render such data misleading. In the example in Figure 6-12, the interface may not reveal enough content to allow you to infer the real level of interest from the number of clicks. (Weight such input accordingly.)

Figure 6-12. By clicking “more,” a reader may be indicating a higher level of interest in this review on Yahoo! Local. (Or a click may simply indicate that the excerpt wasn’t generous enough—tread carefully.)

One of the very best indicators of interest in an entity is the amount of conversation, rebuttal, and response that it generates. While we’ve cautioned against using activity alone as a reputation input (to the detriment of good quality indicators), we certainly don’t want to imply that conversational activity has no place in your system. Far from it. If an item is a topic of conversation, the item should benefit from that interest.

The operators of some popular websites realize the value of rebuttal mechanisms and have formalized the ability to attach a response to a reputable entity. YouTube’s video responses feature is an example (see Figure 6-13). As with any implicit input, however, be careful; the more your site’s design shows how your system uses those associations, the more tempting it will be for members of your community to misuse them.

Figure 6-13. The video responses on YouTube certainly indicate users’ desire to be associated with popular videos. However, they may not actually indicate any logical thread of association.

The desire to optimize their personal experience (see the section Fulfillment incentives) is often the initial driver for many users to go through the effort required to provide input to a reputation system. For example, if you tell an application what your favorite music is, it can customize your Internet radio station, making it worth the effort to teach the application your preferences. The effort required to do this also provides a wonderful side effect: it generates voluminous and accurate input into aggregated community ratings.

Personalization roll-ups are stored on a per-user basis and generally consist of preference information that is not shared publicly. Often these reputations are attached to very fine-grained contexts derived from metadata attached to the input targets and therefore can be surfaced, in aggregate, to the public (see Figure 6-15). For example, a song by the Foo Fighters may be listed in the “alternative” and “rock” music categories.

When a user marks the song as a favorite, the system would increase the personalization reputation for this user for three entities: “Foo Fighters,” “alternative,” and “rock.” Personalization reputation can require a lot of storage, so plan accordingly, but the benefits to the user experience, and your product offering, may make it well worth the investment. See Table 6-1.

Figure 6-15. Netflix uses your movie preferences to generate recommendations for other movies that you might want to watch. It also averages your ratings against other movies you’ve rated in that category, or by that director, or….

Generating aggregated community ratings is the process of collecting normalized numerical ratings from multiple sources and merging them into a single score, often an average or a percentage of the total, as in Figure 6-16. See Table 6-2.

Figure 6-16. Recommendations work best when they’re personalized, but how do you help someone who hasn’t yet stated any preferences? You average the opinions of those who have.

Participation points are typically a kind of karma in which users accumulate varying amounts of publicly displayable points for taking various actions in an application. Many people see these points as a strong incentive to drive participation and the creation of content. But remember, using points as the only motivation for user actions can push out desirable contributions in favor of lower-quality content that users can submit quickly and easily (see First-mover effects). Also see Leaderboards Considered Harmful for a discussion of the challenges associated with competitive displays of participation points.

Participation points karma is a good example of a pattern in which the inputs (various, often trivial, user actions) don’t match the process of reputation generation (accumulating weighted point values) or the output (named levels or raw score); see Tables 6-3 and 6-4.

Points as currency

Point systems are increasingly being used as game currencies. Social games offered by developers such as Zynga generate participation points that users can spend on special benefits in the game, such as unique items or power-ups that improve the experience of the game. (See Figure 6-17.) Such systems have exploded with the introduction of the ability to purchase the points for real money.

Figure 6-17. Many social games, such as Mafia Wars by Zynga, earn revenue by selling points. These points can be used to accelerate game progress or to purchase vanity items.

If you consider any points-as-currency scheme, keep in mind that because the points reflect (and may even be exchangeable for) real money, such schemes place the motivations for using your application further from altruism and more in the range of a commercial driver.

Even if you don’t officially offer the points for sale and your application allows users to spend them only on virtual items in the game, a commercial market may still arise for them. A good historical example of this kind of aftermarket is the sale of game characters for popular online multiplayer games, such as World of Warcraft. Character levels in a game represent participation or experience points, which in turn represent real investments of time and/or money. For more than a decade, people have been power-leveling game characters and selling them on eBay for thousands of dollars.

We recommend against turning reputation points into a currency of any kind unless your application is a game and it is central to your business goals. More discussion of online economies and how they interact with reputation systems is beyond the scope of this book, but an ever-increasing amount of literature on the topic of real-money trading (RMT) is readily available on the Internet.

Compound community claims reflect multiple separate, but related, aggregated claims about a single target and include patterns such as reviews and rated message board posts. But the power of attaching compound inputs of different types from multiple sources lets users understand multiple facets of an object’s reputation.

For example, ConsumerReports.org generates two sets of reputation for objects: the scores generated as a result of the tests and criteria set forth in the labs, and the average user ratings and comments provided by customers on the website. (See Figure 6-18.) These scores can be displayed side by side to allow the site’s users to evaluate a product both on numerous standard measures and on untested and unmeasured criteria. For example, user comments on front-loading clothes washers often mention odors, because former users of top-loading washers don’t necessarily know that a front-loading machine needs to be hand-dried after every load. This kind of subtle feedback can’t be captured in strictly quantitative measures.

Figure 6-18. Consumer Reports combines ratings from external sources, editorial staff, and user reviews to provide a rich reputation page for products and services.

Though compound community claims can be built from diverse inputs from multiple sources, the ratings-and-reviews pattern is well established and deserves special comment here (see Table 6-5). Asking a user to create a multipart review is a very heavyweight activity—it takes time to compose a thoughtful contribution. Users’ time is scarce, and research at Yahoo! and elsewhere has shown that users often abandon the process if extra steps are required, such as logging in, registration for new users, or multiple screens of input.

Even if it’s necessary for business reasons, these barriers to entry will significantly increase the abandon rate for your review creation process. People need a good reason to take time out of their day to create a complex review. Be sure to understand your model (see the section Incentives for User Participation, Quality, and Moderation) and the effects it may have on the tone and quality of your content. For an example of the effects of incentive on compound community claims, see Friendship incentives.

What happens when you want to make a value judgment about a user who’s new to your application? Is there an alternative to the general axiom that “no participation equals no trust”? In many scenarios, you need an inferred reputation score—a lower-confidence number that can be used to help make low-risk decisions about a user’s trustworthiness until she can establish an application-specific karma score. (See Figure 6-19.)

Figure 6-19. If a user comes to your application “cold” (with no prior history of interacting with it), you may be able to infer much information about him from external contexts.

In a web application, proxy reputations may be available even for users who have never created an object, posted a comment, or clicked a single thumb-up. The user’s browser possesses session cookies that can hold simple activity counters even for logged-out users; the user is connected through an IP address that can have a reputation of its own (if it was recently or repeatedly used by a known abuser); and finally the user may have an active history with a related product that could be considered in a proxy reputation.

Remembering that the best karma is positive karma (see the next section Practitioner’s Tips: Negative Public Karma), when an otherwise unknown user evaluates an object in your system and you want to weight the user’s input, you can use the inferences from weak reputations to boost the user’s reputation from 0 to a reasonable fraction (for example, up to 25%) of the maximum value.

A weak karma score should be used only temporarily while a user is establishing robust karma, and, because it is a weak indicator, it should provide a diminishing share of the eventual, final score. (The share of karma provided by inferred karma should diminish as more trustworthy inputs become available to replace it.) One weighting method is to make the inferred share a bonus on top of the total score (the total can exceed 100%) and then clamp the value to 100% at the end. See Table 6-6.