LIES, DAMN LIES, AND METRICS

Use Metrics to Build a Culture of Accountability and Customer Obsession

Gentlemen, we are going to relentlessly chase perfection, knowing full well we will not catch it, because nothing is perfect. But we are going to relentlessly chase it because in the process we will catch excellence. I am not remotely interested in just being good.

—Vince Lombardi

I deliver scores of keynotes and presentations in meetings with business executive teams. These meetings cover broad territory and topics, but I always try to emphasize a few key points: the importance of patience, and that transformation is as much about your personal habits and beliefs as it is about changing your organization. I stress that Amazon’s playbook is a system based off principles. It is not based on a set of steps or a single linear approach.

Inevitably, someone will raise his or her hand and ask me to pick the most important play in Amazon’s playbook. I understand the need to give a starting point. So while explaining that “it’s a system,” there is an answer to the question “What’s the one thing to get me started?” The critical fundamental, the “championship habit” above all else is this: metrics.

IN GOD WE TRUST. ALL OTHERS MUST BRING DATA

In 2004, I attended a senior executive team meeting at Amazon that just so happened to coincide with the Salesforce initial public offering (IPO). During the meeting, one of the other executive team members casually commented that Salesforce was the world’s largest customer relationship management (CRM) technology company.

Big mistake.

“We are the world’s largest CRM company!” a senior Amazon leader (guess who) yelled in response.

Like a CRM company, Amazon is obsessed with managing and analyzing the data on customer interactions to improve its relationships with them. And it is doing it on a much bigger scale than Salesforce. The digital nature of Amazon’s business and its focus on collecting obscene amounts of data make it significantly different than the average e-commerce company. Metrics is seamlessly woven into the culture of Amazon.

W. Edwards Deming’s famous quote is a mantra: “In God we trust. All others must bring data.” Teams spend as much or more time defining and agreeing on how to measure a new feature, service, or product as they do designing the feature itself. Weeks are built into project management schedules for the consideration of an operation’s inputs and outputs; to identify what data might be needed to run that operation; and to understand its complex internal workings. Metrics are the answer. Metrics are the question. Metrics are the vehicle to relentlessly pursue root cause.

“DID MY CUSTOMERS HAVE A GOOD DAY TODAY?”

At Amazon, having a balanced, well-engineered scorecard of metrics that is consistently reviewed day over day, week over week, provides deep insights into what works and what doesn’t. It also places sole responsibility for success and failure on you as a leader.

Repeatable, consistent performance reflected in metrics is the gold standard for success. Without access to a consistent set of metrics, an Amazon leader would be flying blind, and such risky behavior is not acceptable at the company. Real-time metrics are the lifeblood flowing through the veins of Amazon. Real data and real insights from the customer experience are used continually to answer the question, “Did my customers have a good day today?” If your metrics are in place, if they are in real time, and your team members and processes use them, this question yields a simple yes or no answer.

It takes foresight to lead by the numbers correctly. You must embed real-time metrics from the very start of a program because they are nearly impossible to retrofit. The Amazon experience shows us that the single biggest opportunity for companies operating today is to completely rethink their concept of metrics. Most companies use what’s called “batch architecture” to record large sets of transactions or other quantitative updates and to process them periodically (daily or weekly is typical). Batch architecture is very last century. Today, you need real-time data, real-time monitoring, and real-time alarms when trouble is brewing—not lag-time metrics that hide the real issues for 24 hours or longer. Your business should operate like a nuclear reactor. If a problem arises, you need to be aware of it immediately.

SPEND MORE TIME DESIGNING YOUR METRICS

Think about where time is allocated at a company at the executive and management levels—lots of time spent on budgeting, financial reviews, HR, and compliance, and so on. These are important. But do they really help serve customers, drive operational excellence, or spark ideas for innovation? Here are three quick steps for designing your metrics:

1.   Spend time talking about what your metrics should be.

2.   Have meetings that use those metrics to constantly review your business. Always look for root causes. Turn metrics to action.

3.   Get as granular and real-time data as possible.

METRICS IS A VERB

Having great metrics is ammunition, but the wars are fought in the daily combat of using metrics to drive to perfection. The theater for these battles is meetings we called “metrics meetings.” The entire operational rhythm of life at Amazon is a set of weekly metrics meetings. Typically starting with the lowest-level services and capabilities and building up to the worldwide senior meetings at the end of the week, an operational leader at Amazon attends a series of metrics meetings over the course of a week. Every week.

Although they are called “metrics meetings,” they are really meetings of shared accountability where the “owners” of related metrics discuss the most recent trends and issues in their metrics. Leaders are expected to know the details of their business, to be vocally self-critical, and to discuss plans or progress for improvement. Although successes might be recognized, the general modus operandi is to “celebrate for a nanosecond” and then focus on the issues and errors in the business. Even when things are generally good, they’re never perfect.

Meetings are organized around key services, processes, and groups of capabilities. For example, I ran the Marketplace metrics meetings, even though from an organizational standpoint, only a subset was part of my organization. I would include leaders from key capabilities affecting the Marketplace business, such as the catalog team, order pipeline team, and payment and fraud team.

The finance partners at Amazon play a key role in making sure all metrics, not just financial metrics, have been prepared, and they partner with the functional leader to take notes, assign responsibilities, and drive results and improvements to be delivered. The finance partners are critical in keeping the meetings “honest” and ensuring that people deliver results. I think of them as the independent accountability officers ensuring that the metrics meetings are doing their job of defining and reporting on progress to improve our performance.

INSTRUMENTATION: REAL-TIME, FINE-GRAINED DATA

Once a team’s metrics and SLAs are in place, the focus turns to data collection that will inform those metrics and SLAs. At Amazon, there are very specific standards for the quality and type of data a team should collect. Amazon’s executive team refers to those standards as “instrumentation.”

Expectations on data collection—or instrumentation—at Amazon are twofold: First, data should be fine-grained in nature. You can always summarize and aggregate data, but you can’t go back and derive more detail from a data set. Second, that data should be available in real time. You can always batch data or slow it down, but you can’t speed it up. Design for no time lags and no batch systems.

There are many reasons this is important. Let’s say a grocery company uses a refrigerated storage bin to keep its fruits and vegetables fresh. Suddenly, over the course of a day, all the lettuce goes bad. If the company has been collecting fine-grained data—for example, any changes in temperature or pressure and their time stamps—that grocery company might actually be able to figure out what it was that caused the lettuce to go bad. Otherwise, they’re stuck wondering about the possible variables and causes in that situation.

Of course, it’s not possible to collect fine-grained real-time data in every situation. There are limitations to the nature of the data that you’re able to collect in a specific situation, but at Amazon it’s expected that you will work vigorously to achieve instrumentation.

Finally, never stop reevaluating and building metrics. Frequently check to make sure that your metrics are still explicitly linked to goals, and don’t hesitate to change metrics when they stop driving change in your organization.

Why?

Because good metrics create good processes that minimize bureaucracy.

Figure 31.1 Metrics: The Rhythm of the Business

Figure 31.2 Metrics Is a Verb