Chapter 12. Monitoring and Evaluating Performance

Throughout this book, you have read of challenges you face when developing and implementing marketing-related activities in the public sector. However, many pale in comparison to the topic of evaluation, not only because it is one of the least understood but also because it is one of the most requested and hotly debated. You have probably witnessed some of the following states of confusion and dilemmas in your own agency:

• I don’t understand how you know whether marketing contributed to what happened compared to other factors. We don’t know whether our public meetings with citizens created the reduction in crime in that neighborhood, but that’s what the boss wants to know.

• How can we possibly justify thousand of dollars to report on what good our marketing expenditures did? We’re already getting heat for how much we spent on the campaign alone.

• To be honest, I’m worried about the results. I’ve got this sinking feeling that this didn’t do any good and that we won’t see that we moved the needle one bit. Then what?

By the conclusion of this chapter, you will have insights on how to address these challenges and more importantly a model to share with others next time you face these issues, one that was successfully applied in our opening story.

Opening Story: An Environmental Behavior Index—Helping to Quantify What Happened and Decide What to Do Next

Challenges

King County, Washington and its Department of Natural Resources and Parks are committed to protecting the environment and conserving taxpayers’ dollars. The questions administrators decided to tackle in the fall of 2004 were, “How do we do both?” and “How do we know how we’re doing?”

At that time, the department was allocating resources to several dozen programs designed to influence environmentally friendly yard care, recycling, waste disposal, and purchasing practices among county residents. The goal was to develop a common evaluation tool that would measure current behavior levels for each of the programs being promoted, with an interest in determining which behaviors had the greatest market opportunities for growth (increased participation) and, when overlaid with environmental impact, should be considered for continued or increased resources. The tool was also meant to help the Department track progress over time of the various behaviors to support its performance management system.

Strategies

In the spring of 2005, a telephone survey of 1,001 county residents was conducted to assess current levels of participation in 29 key environmental behaviors being promoted by the Department. For each of these behaviors, respondents were asked a series of questions that resulted in their household being categorized as one of the following:

Bright Green—Do the desired behavior all or most of the time

Light Green—Do the desired behavior only some of the time

Yellow—Do not do the desired behavior but have thought about it

Brown—Do not do the desired behavior and are not considering it

Grey—Do not know about the behavior or what their household is doing

White—Does not apply (e.g., don’t have a yard or lawn)

Programs were then ranked (from 1 to 29) based on the greatest number of Light Green and Yellow households, those viewed as most likely and most open to change—to conduct the behavior on a regular basis, at the desired level (see Figure 12.1).

Figure 12.1. As indicated by the percentage of those in the light green and yellow categories on this bar chart, only 12 percent of households appear to be “available/open” to increased regular disposal of grass clippings compared with 42 percent for reducing the size of their lawn.

Rewards

Results of the survey have helped the Department in several important ways.

First, as hoped, it helped with prioritizing programs for resource allocation, with findings indicating that for 15 of the 29 behaviors, at least 20 percent of households in the county were either considering doing this behavior (the Yellows) or were doing it some but not all of the time (the Light Greens). Examples of markets of opportunity included increased use of energy-saving light bulbs (with 56 percent light green or yellow), proper washing of cars (33 percent), and giving experience gifts rather than “stuff” to reduce waste (23 percent). The environmental impact of each behavior, funding sources, operational feasibility, and the need for strategies beyond education were also to be considered when changing funding allocations among programs.

Second, it provided valuable data for programs when developing marketing plans. Target markets (Light Greens and/or Yellows) for specific behaviors can be described in terms of unique demographic, geographic, and related-behavior variables—data that was collected for respondents and their households. The Light Greens for using only natural or organic fertilizers, for example, were more likely to be college-educated males with children six and under living in the household. This then can assist in the development of more targeted, effective, and efficient marketing strategies, ones most likely to influence intended behavior change.

Rankings also assisted in developing public communications. Press releases, brochures, and Web sites reinforced behaviors that appeared to be cultural norms (e.g., 81 percent indicating they used recycling containers at home), focused on those with low awareness (e.g., with 67 percent indicating they hadn’t heard of the EnviroStars program recognizing environmentally friendly businesses), and corrected apparent confusions and misunderstandings (e.g., prescription drugs should not be put down the toilet or sink but should be put in the garbage or returned to a pharmacy that accepts used medications). Information from the Environmental Behavior Index is also reported to the public as part of the Department’s performance management report.

Finally, this first survey now serves as an outcome-based performance measurement baseline for monitoring and tracking changes over time. Knowing these current levels, managers can set goals for numbers of households to increase behaviors, and then outcomes can be measured against these goals on a periodic basis. Levels of change can then be calculated, and costs (in taxpayer dollars) supporting this change can be analyzed relative to outcomes and anticipated environmental benefits.

Measuring Marketing Performance

Measuring marketing performance is one of the most complex, even frustrating, of all marketing tasks. It is often expected and then questioned; it can be applauded and then forgotten.

In an ideal scenario, you can step forward and report on the return on your marketing investment of time, money, and other resources to develop and carry out marketing programs—an uncommon situation indeed.

This chapter is intended to help by outlining components of an evaluation plan, posed in the form of questions you’ll want to answer in a sequential way, starting with the toughest one:

• Why are you conducting this measurement and for whom?

• What will you be measuring?

• How will you conduct these measurements?

• When will these measurements be taken?

• How much will it cost?

Your evaluation plan is developed before your final marketing implementation plan as it will need to be covered in your budget and noted in your action items.

One distinction is important up front: the difference between the term “monitoring” and the term “evaluation.”

Monitoring generally refers to measurements that are conducted sometime after your new program and/or promotional campaign has been launched but before it is completed. Often you do this to determine whether you need to make mid-course corrections that will ensure that you reach your ultimate marketing goals (e.g., 50 percent of businesses filing their tax returns online). You may want to analyze whether you are reaching your target audience and how they are responding. Of interest may be how management and staff are viewing the effort, whether your competition has reacted, and what to do about it.

An evaluation is a measurement and final report on what happened, answering the bottom-line questions: Did you reach your goals? Can you link outcomes with program elements? Were you on time and on budget? How do costs stack up against benefits? Were there any unintended consequences that will need to be considered going forward—maybe even handled now (e.g., increase in cigarette butt littering as a result of restrictions on smoking at entrances to public facilities)? Which program elements worked well to support goals? Which ones didn’t pull their weight? Was there anything missing? What will you do different next time, if there is a next time?¹

Why Are You Conducting This Measurement and for Whom?

First, determine why and for whom you are conducting this measurement. There are several possibilities:

• To fulfill an expectation for reporting on campaign outcomes

• To fulfill funding or grant requirements

• To make changes mid-course or before a campaign rollout

• To do better next time

• To get more or continued funding

As may be apparent, methodologies and reports will vary depending on your answer to this first question. If you are conducting this measurement to fulfill some grant requirement, you will be guided by pre-established specifications. If, on the other hand, your audience is primarily internal and your main purpose is to do better next time, you will want to be diligent in your evaluation of each campaign component and interested in understanding what contributed to your outcomes and what didn’t, as was the case in the following example.

In preparation for a campaign to reduce chewing gum litter in the U.K. scheduled for the spring of 2006, a four-week campaign pilot was conducted in 2005 in three local towns. Pilot campaigns used catchy advertisements, which included the message “Thanks for binning your gum when you’re done,” offered alternative disposal solutions such as gum pouches, and highlighted the threat of fines (see Figure 12.2). These pilot strategies had been based on formative research that identified reasons why some people drop gum (i.e., they didn’t know it was a problem) and what might motivate them to change behavior (i.e., the threat of enforcement). Results from one of the pilots indicated that gum litter had been reduced by 80 percent. The Minister for Environmental Quality believed “the pilots last year gave us a good idea of what worked and what didn’t.” Information and experience gained informed a wider rollout and assisted in the development of a guide for local authorities on “how to tackle this sticky problem.”²

Figure 12.2. Campaign to reduce chewing gum littering in the U.K.³

What Are You Measuring?

What you will measure to achieve your research purpose is likely to fall into one or more of three categories: outputs, outcomes, and/or impacts. As you will read, efforts and rigor required vary significantly by category.

Output Measures

Perhaps the easiest and most straightforward measures are those thought of as marketing outputs, sometimes also referred to as process measures. As the label implies, these measures are ones that reflect levels of marketing activities, as quantifiable as possible. Most fall in the category of promotional efforts because the word “output” focuses on messages sent from your agency to the marketplace.

Common output measures, usually available in records and databases, include:

• Number of materials distributed (e.g., number of brochures on identity theft handed out at a city’s police department and community centers)

• Reach and frequency of advertising (e.g., number of citizens estimated to have been exposed to state-sponsored radio spots and busboards regarding identify theft and the number of times they would have been exposed to them)

• Number of impressions from other communication channels (e.g., number of customers at a grocery store who would be exposed to messages on grocery bags regarding identify theft)

• Mentions and airtime in the news media and anticipated numbers of people reached by them (e.g., number of minutes of news coverage on a local television station and numbers of people typically viewing that program)

• Numbers of special events held and anticipated numbers of people reached at these events (e.g., number of presentations made by police officers at local community groups such as Rotary Clubs and numbers of people attending)

• Resources expended noting time and money (e.g., amount of money spent to develop and implement identify theft campaign elements and amount of time staff spent managing the campaign)

• Other marketing-related activities with some exposure among target audiences (e.g., number of links you were able to establish on related Web sites such as those of local police departments)

Notice that output measures say nothing about customer or citizen response to your efforts. They only reflect your “flurry of activities”—what campaign elements you put out there, what it cost you, and the estimated number of citizens (ideally, target audience members) exposed to them.

Outcome Measures

What most of you and those you work with also want to know is how citizens responded (outcomes) to what you did (outputs). Were your messages and materials even noticed? Was anyone listening to your ad on the radio or your speaker at special events? Were you able to create partnerships or generate contributions to support funding? Did these activities change levels of knowledge, attitudes, beliefs, or levels of citizen satisfaction? Did they inspire people to find out more? Most importantly, did they influence citizen action, reflected in increased purchases, participation, compliance, or behavior change? In each case, your focus is on answering the challenge “So what happened?”

If you are following a systematic marketing planning process, you’ll find this identification of outcome measures easier because likely factors to be measured are rooted in your goal and objective statements.

Consider Maine’s Think Blue campaign to protect water quality. Formative research conducted before a 2004 campaign revealed that the majority of Mainers had no idea what a watershed is, where storm water goes, or what is polluting the water in neighborhoods. Campaign outputs included radio and television ads, a Web site, a print piece, hosting or supporting local events, and forming partnerships. But their definition of success was to be determined by outcomes based on results from surveys conducted with target audiences. Results were encouraging:

• Campaign awareness—Two months after the end of the campaign, of those surveyed, 14.4 percent of Maine adults remembered the ads and the message; 8.7 percent said the ads were about storm water runoff.

• Changes in knowledge, attitudes, or beliefs—Awareness that soil erosion is a source of water pollution increased from near zero to 6 percent; concern with water quality increased by 8 percent.

• Changes in behavior or behavior intent—26 percent of the adult Maine population said they had taken or were likely to take one or more of the recommended actions to reduce storm water pollution.

• Creating partners or campaign contributions—Multiple agencies and organizations in the state contributed funding to the media campaign and to support special events—leveraging the Department’s money, staff time, and resources.

• Increases in customer satisfaction—Although this outcome was not measured for this campaign, an example would be if they included a question in their survey regarding citizen satisfaction with what the state was now doing to protect water quality.

As with the King County Environmental Behavior Index mentioned earlier in this chapter, these outcomes can now assist program managers when setting goals and objectives for future campaign outcomes.

Impact Measures

You will probably get a thankful nod from administrators when you provide a recap of your marketing program activities and expenditures (outputs). When you then go on to describe and quantify citizen response to your efforts (outcomes), you will most likely capture and hold their attention. If and when you are able to take it to the next level and report on the actual impact these citizen actions had on social, economic, and/or environmental conditions, however, they will applaud your efforts—as they should.

Notice in the following list of potential impacts from your marketing efforts the more rigorous questions that you will need to answer:

• Cost savings—How much money does the E-ZPass program save each year in operational costs?

• Increases in net revenues—Has the increase in admission revenues at the Louvre paid back the costs for improvement yet? When might it?

• Funding and levies approved—Do we know whether the community presentations the police department gave actually contributed to the levy passing?

• Lives saved—We understand there has been an increase in life vest usage. But has there been a decrease in drownings or near drownings as a result?

• Diseases prevented—How many cases of HIV/AIDS were prevented as a result of the increases in testing we achieved with our target audience?

• Water quality improved—Are the salmon healthier because the use of chemical lawn fertilizers has decreased in homes on the waterfront?

• Water supply increased—How much more water do we have in the reservoir as a result of the number of low-flow toilets we subsidized?

• Air quality improved—Are there fewer asthma attacks in the county as a result of the decrease in woodburning fires?

• Landfill reduced—How many acres of land did we preserve as a result of our 25 percent increase in recycling?

• Wildlife and habitats protected—Are there more baby crabs in the waters of Maine as a result of citizen actions to reduce storm water pollution?

• Animal cruelty reduced—Are pets living longer and with fewer injuries and diseases as a result of the increased fines in Turin, Italy?

• Crimes prevented—How many fewer assaults do we have in the park since we started blasting classical music over the loudspeakers?

How Will You Measure?

You have three major informational resources to assist you in your efforts to measure outputs, outcomes, and/or impact: internal records and databases, citizen surveys, and scientific/technical surveys.

Internal records and databases are perhaps the most relevant and reliable way to measure and report on outputs such as the number of brochures distributed on how to reduce your electric bill or the number of visitors to the post office who might have seen the notice regarding the availability of passport applications. In some cases, you can also use records and databases to track outcomes, as you would if you wanted to know the number of citizens and businesses filing tax returns online before and after your campaign this past year. With some planning and preparation, you could also use this resource to determine impact, as you would when analyzing any decreases in HIV/AIDS infections and correlating these with increases in testing.

Citizen surveys may be the best and sometimes the only way for you to determine outcomes from your marketing efforts. Your target audience will need to tell you whether they are more informed (e.g., on where to go for more information on property tax increases), whether they are motivated (e.g., are now considering use your port for cargo ships), whether they have changed a belief or attitude (e.g., that the schools need to be remodeled), or whether they indeed changed a behavior (e.g., drove one fewer time a week). As noted in Chapter 11, “Gathering Citizen Data, Inputs and Feedback,” a variety of techniques may be appropriate including surveys conducted by phone, mail, online, or face-to-face. Some may be accomplished most effectively through observation surveys (e.g., watching for increases in pedestrian use of crosswalks).

Scientific or technical surveys may be the only sure method to assess the ultimate impact of your efforts. If you are really charged with reporting back on the difference your efforts made in improving air quality, water quality and supply, the health of fish and wildlife, and/or the value of diverted materials from the landfill, you will need help designing and implementing reliable scientific surveys that not only indicate changes but can also link these changes to your marketing efforts. You will also need this approach when making the case that decreases in diseases (e.g., obesity) can be linked to increases in desired public behaviors (e.g., physical activity).

It should be noted that you will most likely need to provide data from any, if not all, of these sources to those developing financial models (the ultimate measurement tool) for your agency, assisting you and others in a final economic assessment of past and potential future activities.

When Will You Measure?

With a clear sense of why, what, and how you will be measuring your performance, your next decision is when. Options reflect this timing.

Pre and post surveys, as the name implies, are conducted at some point prior to your campaign and then again at some point after it is completed—or at least a phase of it. A state launching a new slogan for attracting tourism, for example, may conduct a pre and post survey to determine any intended changes in attitude towards the state and intent to visit, linked with exposure and proven recall of the campaign. When resources or other factors make both measurements unlikely, a post survey alone might be conducted, which is still useful but more challenging in its ability to link outcomes with outputs.

Longitudinal surveys track and measure changes over a period of time, most often studying changes in a specific population group. The U.S. Department of Labor, for example, conducts a set of longitudinal surveys designed to gather information at multiple points in time on labor market activities, assisting those analyzing unemployment statistics and employment-related activities by geographic areas, age groups, and gender.⁴

Periodic surveys may be used for monitoring as well as evaluation of efforts. One evaluation measure of a state’s litter campaign, for example, may be a scientific litter survey conducted every two years that measures changes in the amount and content of litter on targeted state highways and county roads and then overlays data on location of campaign activities such as road signs.

How Much Will It Cost?

Finally, you identify costs that will be associated with developing and conducting your monitoring and/or evaluation activities to be included in your final marketing budget.

When significant, these will include staff time to manage the process and, if relevant, will certainly include staff time to field the survey (e.g., time to conduct observational surveys on household recycling practices) or conduct the analysis (e.g., time needed to develop financial models).

It is not uncommon that you will end up contracting with an outside research or consulting firm to conduct these measurements. The need for quantifiable results will increase the numbers and therefore the time required to survey citizens, and the need to ensure a lack of bias may lead to a third-party choice.

Summary

Questions to answer when you are developing a plan to monitor and evaluate the performance of your marketing efforts are straightforward. Why, what, how, and when will you be measuring, and how much will it cost? It is the answers that are tough, especially when deciding the “what,” as you will no doubt be torn between the desire to report on the return on your agency’s investment and the need to just get it done; the desire to be accurate and the need to stay within budget; and the desire to learn from your mistakes as you go forward and the need to “look good” today.

Measuring your outputs (marketing activities) is more tedious than complex, usually “just” requiring you to develop a records and database system that will document the nature, quantity, frequency, and exposure of your efforts in the marketplace—most often those promotional in nature.

Measuring the outcomes of these activities in terms of citizen response is a little less certain and more resource intensive, as you strive to measure and then report on responses to your marketing program elements. For these measures, you will typically need to survey citizens to determine changes in intended knowledge, attitudes, and behaviors and use methodologies that can correlate your activities with theirs.

And then there’s the “best of show,” awarded because you measured and reported on the impact your efforts had on your agency’s scorecard—indicators of accountability, including those that are social, economic, and/or environmental in nature. You know the harsh realities of this endeavor. It will take more time, more money, and more stamina, as even the most well-intentioned methodologies will be subject to peer and management scrutiny. Regardless, you are encouraged to make progress in this arena, settling at times for narrowing the focus of your measurement on only a few of your efforts, a pilot, a single geographic area, or a single target audience. Remember, a small step is still a beginning.