11

MEASUREMENT AND METRICS

We think of community measurements assessing three areas. Each area informs about a specific and important part of a brand community’s success. Each area impacts on the success of the other areas. Or, simply put, failure in any one area will harm the others.

1. Community activities: experiences offered to members. These include programs, events, blog posts, trainings, celebrations, rituals, and so on.
2. Community health: Members’ satisfaction that they feel supported, included and accepted.
3. Organization outcomes: Behavior, skill, or resource that supports organizational goals.¹

This chapter will provide principles for how and what to measure. There is too much diversity among brands and communities to share exactly what you should measure at any time. You’ll get to explore what’s right to measure for your priorities.

Measuring Community Activities

Relationships just take time to develop. So, in early stages you can measure the actions to bring a community together instead of results. This can mean tracking actions that provide members value and a space that encourages connection. For example, it may be best to track how many invitations are made and how many events are offered. Leaders often need to offer many events to learn what appeals to members most or even at all.

Early-stage communities often don’t yet create meaningful outcomes because they’re too young. Moreover, even when your community does produce lots of outcomes, it’s possible to lose community strength if insufficient attention is paid to the working relationships within the membership. In both cases, measuring community health separate from outcomes helps leadership work on ensuring durability so any good outcomes grow and persist.

We’ve mentioned several times that although there are many things we can measure quantitatively (motorcycles sold, event participants, contributing members, volunteer hours), all these measures can be manipulated by a community manager. For example, when you give away free pizza, more Oaklanders will show up. The stronger attendance doesn’t mean you’ve got a stronger community. Although this is true, it’s hard to build a strong community without events, places, invitations, and communication to bring people together. So measure what experiences are offered and the actions involved to offer them.

Measuring Community Health

There can be lots of activity (dancing, dumpling eating, or drumming) and not much community health. We can dance and eat around people we don’t feel connected to, known by, or welcomed by. So collecting data that indicate members are connecting is also important. There is no single magical way to do this. A collection of proxy measurements can help us understand community health trends.

Three to five community health metrics are usually all leaders can track to inform them when and where problems need attention. Here are some metrics that in aggregate can help you see whether this is happening:

• Gross size: membership, visitors, attendees, readers.
• Conversion rate: how many people convert from visitors to members.
• Churn: how many members drop out.
• Meaningful engagement: Experiences that indicate participants develop or show mutual concern. What you measure will vary a lot depending on what your community does together. Look for how participants contribute. (Clicks don’t mean much for community.)

To most accurately understand community health, qualitative research is by far the most important. Qualitative research investigates participants’ reports on their own experience (feelings, motivations, perceptions) even though there’s no standard for comparing their responses. Qualitative research often includes open-ended survey questions, focus-group conversations, in-depth interviews, direct observation, and the like.

Quantitative research (surveys with rating scales, for example) may help you recognize trends and surprises. For example, we can notice that more participants report feeling more connected after an event than reported at earlier events. Although we don’t know what any one participant means by “feeling more connected,” we can look for the trend as we create events.

Marcus Graham at Twitch knows that one of the brand’s important competitive advantages is that the top two hundred broadcasters rightly feel connected with Twitch leadership and feel supported connecting with members. Twitch doesn’t need ten thousand more broadcasters looking for free pizza; it wants the top two hundred broadcasters to feel so connected that they never consider leaving. Marcus knows that, at the end of the day, those relationships deliver both customers and growth. So Twitch asks important members how connected they feel.

We’ve seen many after-event surveys that ask about participants’ satisfaction, enjoyment, or effectiveness. There’s nothing wrong with great responses on these questions. However, if you’re seeking to make participants feel connected, welcome, safe, enriched, seen, and/or appreciated, then they may not know what “effective” means for you. Participants’ feeling “satisfied” by the experience may also mean very little if they feel neither connected nor appreciated.

Consider having members rate the following five statements on a Likert scale (from “Absolutely agree” to “Do not agree”) and then track general responses over time so that you can learn how connections evolve among members.

• I believe others here care about me.
• I care about the others in this group.
• I think people here know who I really am.
• I believe that in a tough time, people here will help me.
• There are enough opportunities to connect and get to know others here.

Frankly, we’ve learned that most events are so unsatisfying and poorly thought out that participants are delighted when an event simply avoids being a boring, confusing, noisy, and/or aimless time. This is when your observation is important. You can notice whether, when, and why participants laugh, cry, hug, high-five, smile, cheer, dance, break out into long private conversations, trade contact information, share meals, and anything else that indicates that relationships are forming or deepening.

Twitch’s Marcus Graham has access to participation tracking for over one hundred million users and still believes that measuring participation is always the hardest part. We agree with Marcus, who says that you can often see the importance and power of a community when things go bad (personal communication with Marcus Graham, April 2019). In bad times, we can see who supports one another, and how, when support is needed. But none of us want to create disasters as a way to see our community strength. Still, we need a way to see how things progress before bad times occur.

Measure Relationships

When it comes to community building (not necessarily promotion), we must understand that, in relationships (among friends, with your parents, with neighbors), successful growth is very often hard to measure. We agree with David Brooks’s assertion that “care is primarily qualitative. A community is healthy when relationships are felt deeply, when there are histories of trust, a shared sense of mutual belonging, norms of mutual commitment . . . and real affection from one another’s heart and soul to another”(emphasis ours).²

In chapter 1, we discussed the difficulty measuring qualitative experiences like mother’s love quantitatively. We’re revisiting the example for those who share this chapter as an excerpt and because the limits of qualitative measurements are important to this discussion. To understand how mutual concern (authentic community relationships) remains qualitative, and quantitatively elusive, consider again mother’s love. We hope that you have experienced mother’s love. With few exceptions, mothers care very deeply about the welfare of their children. Despite the fact that most of us believe that mother’s love is critically important for many rewarding, satisfying, and enduring experiences in our families, it will be either hard or impossible for most of us to prove quantitatively that mother’s love has grown or eroded in the past five years.

Measuring shared meals, phone calls, holiday visits, or any other activities may help you understand the relationship better. Looking at trends may inform your understanding of the relationship trend, but of course these measures will never accurately assess mother’s love in your family (or ours), or even come close to it.

In relationship, the value or impact of each action, investment, or choice is hard to assess externally. The importance lives in individuals’ experience. If you make the mistake of depending on external metrics to judge internal experience, then you’re at real risk of being distracted by surrogation. Surrogation is the tendency for people “to confuse what’s being measured for the metric being used.”³

Wells Fargo provided a clear example of surrogation when it measured building long-lasting customer relationships by means of “cross-selling” metrics. Cross-selling for Wells Fargo meant that employees would sell more products to current customers. The result of using this metric, however, was that employees increased the number of new accounts by opening 3.5 million new customer accounts without customers’ consent. Employees scored high on the metric, but they missed any truly helpful business goal. The company continues to pay for the mistake in at least a billion dollars in litigation costs and lost brand integrity.⁴

Community strategies can fall prey to surrogation, too. For example, your strategy may be to build deeper relationships between customers and the business, but if the only metric you use to measure progress is the number of messages sent from your employees to customers, your team may begin to send pointless messages to improve their numbers. This is especially true if their performance evaluation or even compensation is tied to these scores.

We all want to know when we’re succeeding, and measurements that document progress are valuable to share with stakeholders. The difficulty comes when people who don’t understand how communities work and succeed choose unhelpful metrics and then judge the success on those metrics using meaningless or misleading data.

At the end of the day, we don’t think it is very important to measure mutual concern quantitatively. You know members care for one another when you see it show up as action. This can include volunteering, donating, attending events, or personally checking in on one another. This is often something that outsiders cannot see or feel.

Regularly Survey Self-Reported Measures of Belonging

We can and should collect subjective (qualitative) measurements of belonging. Learn whether your efforts are working or need a change by surveying whether and how much your members feel connected. No one makes all the right decisions from the beginning. Discover what will work by getting the feedback.

As an example, when discussing Airbnb’s community investment, Guy Michlin, head of Airbnb host success and community, explained that the company both wants to know whether the events are helping build community and wants to make the research as burden free for participants as possible. Airbnb knows it’s critically important that its hosts around the world remain connected to the company, because they provide all the experiences and accommodations for customers. So Airbnb creates dozens of events around the world that facilitate hosts’ connecting with one another and sharing best practices and learned lessons. And after much research, Guy’s team has settled on asking hosts two qualitative questions that tell them enough to guide the company’s work (personal communication with Guy Michlin, July 2019):

1. How connected do you feel to other hosts?
2. How supported do you feel by Airbnb?

Measuring Outcomes for a Brand

The first step to measuring outcomes for your brand is to determine the outcomes you want. What you measure will of course be informed by the goals you want the community to support. In part 1, we listed seven organizational goals that communities can support. All members can’t deliver on all organizational goals every day. You’ll have to choose priorities.

Measuring community brand support success can become a very sophisticated and resource-intensive project. We’ll just introduce some fundamental terms and tools. You can use them to help choose what and how you will measure for your goals.

When getting started, pick one of the goal areas and as few metrics within that area as possible that will still inform you that value is being delivered. You can always change your focus later when experience reveals new wisdom. Tracking and responding to too many metrics distracts and pulls attention from the actual work of building a community to critical mass stage. You can imagine a community manager at Acme customer service brand focusing on improving the outcomes for all of the following areas to reinforce customer support alone:

• Reduce time lag to customer questions
• Increase in customer satisfaction scores
• Improvement in individual ratings of response quality
• Number of provided answers
• Reduction in new incoming questions
• Reduction in other support resources used
• Lowered staff time spent in customer support
• Increased product use

All of these are important, but monitoring every item on this list would make anyone dizzy and frustrated. Just measuring these outcomes could (and should) require an experienced full-time staff, and interpreting the data would be an additional challenge. Most people don’t have the time or resources to monitor so much while building a community. Without that staff support, the list would be more distraction than tool.

Find a proxy for the impact you seek (data entered, volunteer days, people aware of your work). Measure things that your community can deliver and control. If, for example, you measure data entered by volunteers, the numbers can mislead if the community has no control over gathering the source data.

The following are common helpful metrics to review among community members:

• Customer retention. This may be the most important metric to watch because keeping a customer is five to twenty-five times cheaper than gaining a new one. This metric measures who returns to repeatedly purchase/experience your brand.⁵
• Number of new customers. The number of new customers in any given time period.
• Customer satisfaction rating (CSAT). This is a measure of how satisfied customers are with your brand experience. Typically, this is rated on a Likert scale from “Very dissatisfied” to “Very satisfied.”
• Net promoter score (NPS). This is a proxy measurement for future growth that compares the number of brand promoters to the number of detractors.
• Customer lifetime value (CLTV). This is the prediction of the net profit of an entire relationship with any customer.
• Change in sales revenue. This is the increase or decrease of revenue in any given time period.
• Cost savings. This measures costs decreased because of strategy changes. For brand community leaders in customer service and talent retention, this may be the only metric they care about when considering community investment costs. This metric is often unimportant for marketing goals.

We’ve heard too many times that many brand executives will make community investment decisions only using measures that indicate what investments directly serve organizational goals (outcomes). There is no problem with measuring to ensure an investment is succeeding. The problem comes when focus on one area leads to decline in the other areas that shape outcomes.

Several internationally influential practitioners in world-famous, well-resourced companies shared that measuring the importance of and ROI for brand community efforts remains the most difficult part of their community building. Large companies hire PhD researchers to handle this. Most organizations lack the resources to hire advanced research teams or the capacity to track sufficient member activity.

Advocating for Community

The biggest challenge shared among the brand community executives we interviewed was advocating for brand community investment within their organizations. Such advocacy is difficult when success depends on members’ internal experiences, yet organizational leaders demand quantified external metrics. Even among the leadership of global brand communities that clearly serve brand goals (including Etsy, Twitch, Yelp, Patagonia, and Airbnb), executives believe there still remain unmeasured benefits and value to their organizations.

Obviously, growing member numbers is good for audience growth. What else can help advocacy?

Compare Members and Nonmembers

One strategy is to compare the behavior and outcomes of members (variable) and nonmembers (control). Unfortunately, this strategy has an inherent problem in that the observed variable (members) has an inherent bias toward activity. For example, Sephora learned that its community members spend many times more than nonmembers, but we don’t know whether big spenders are just more attracted to the brand community. It’s possible that they may outspend the same without membership engagement.

Of course, this seems unlikely because an enthusiastic customer, engaged and well informed on exciting opportunities, seems more likely to buy more. This makes intuitive sense, but we simply don’t know.

To account for this, you can use a “synthetic control.” This means you find people who aren’t participating in your community and also have profiles similar to those of your members. You may have to compare specific members and nonmembers (e.g., compare a Seattle-based author member and a Seattlebased author nonmember). Then you can track any outcome differences you like (e.g., purchases, volunteering, advocacy).

By any standard, this is very hard to do, and to get any good data will almost certainly require the full-time attention of at least one highly skilled person (likely with a graduate degree in research). We only know of well-resourced organizations that do this.

You can find more discussion on goal-specific measurement in appendix A: Brand Goal Practices.

Consider the Counterfuture

Many brands succeed just fine without brand community investment. We support this if the community will just become a distraction. Some organizations, in contrast, understand that a strong brand community is core to their competitiveness or survival (e.g., Etsy, Yelp, Salesforce, and Twitch).

You can think through and articulate the predictable scenario—the counterfuture—if your brand doesn’t invest in community. Start by answering questions like these:

How will we attract the best talent? (by bidding war?)

How will we gain access to innovative input?

How will we distinguish ourselves from competitors?

How will we educate the market on sophisticated innovations?

How will customers learn to use our product without our help every day?