Before we start talking about the types of metrics to collect and the analytical tools to use, we need to define the two terms to solidify our understanding of the distinction.

Analytics and metrics work hand-in-hand: first we come up with the right metrics to track, then zoom out and examine the relationships between the collected data.

There is no “one magic metric” that every PM needs to flag for collection. Collect too little data, and you run the risk of missing patterns that lie below the tip of the iceberg. Collect too much data, and you could lose time parsing through all the information (especially if the repository is sizeable in nature). At a high level, we can hypothesize the general questions we want to answer through our metrics:

Once this is set, huddle with the team and management to think through the success criteria. Is it to maximize revenue? Grow the user base exponentially? Sunset (remove) features that aren’t well performing or frequently used? Brainstorming will align goals, and you can form the metrics around the main product objectives. Metrics that guide key decision making are called key performance indicators (KPIs). Companies will tie KPIs to financial, business, and other success criteria, and setting a mix of realistic and stretch goals for the KPI goals keeps the team focused and efficient.

Metrics indicate success or failure in whatever you’re measuring, but without categorizing them properly, you could end up mixing one type of metric with another, leading to messy final insights and scrambled reports. To cover all of the bases when choosing trackable metrics, answer the following questions in the six categories as you put together a data analytics strategy: financial, business, product, process, people, and user. Use the categories and questions to frame the thinking when making the conscious decision on what to track versus what to ignore.

Financial Metrics

Business Metrics

Product Metrics

Process Metrics

People Metrics

User Metrics

In the search to find the perfect mix of metrics to harvest, there are a baseline set of data points that need to be mentioned. Every product will not find the following list of metrics relevant, but in general, they’ll come up over and over again in product discussions throughout a PM’s career, and knowing the meaning behind the name and/or acronym is handy.

A/B testing (also known as split testing ) is an effective method to measure performance of two websites by comparing one version (A) to another (B). Often used to measure conversion and engagement, A/B testing is extremely helpful in finding out which variation of a particular page resonates most with the end user. To set up an A/B test, use a tool (Optimizely, Google Analytics ) to inject a snippet of Javascript code, then create two distinct webpage versions, and run the different pages on live users. If you notice an uptick in one experiment versus another, you can swap out the old page with the new and continue tweaking the elements on the page to maximize conversion.

In Figure 8-1, the first Apple.com webpage (A) uses the “Say hello to the future” header.

In Figure 8-2, the second webpage (B) uses “the future is crystal clear” as the header. In this A/B test, Apple can surface both pages to a statistically representative sample of users, and record the header that results in more iPhone X sales.

For products and companies with an established user base, A/B testing may only scratch the surface since only two variants of the webpage can be tested at one time. With multivariate testing, we can test as many possible permutations of the website as we want, granted we have the user base to split the pages amongst.

In multivariate testing , the original page is split into smaller components. In the page shown in Figure 8-3, we can mark the top navigation bar, the title / header, and hero image all as disparate components to use for our multivariate experiment. Now, let’s say we want to test the following areas by:

With the components identified, we can tell that there are 12 possible combinations possible (2 X 3 X 2 = 12). From this point on, we can use industry tools to push out 12 versions of the page to end users and track conversion and other related metrics.

To me, analytics have always been an unlock of stories that otherwise would be untold. Often, analytics provide the sole answer for issues that drive reduced efficiencies, non-optimal processes, or fractional revenue and margin potential for organizations. In today’s landscape, where companies are rigged with antiquated technologies and solutions, I continue to see analytics unveiling significant opportunities, with projects, programs, and investments all rallying around the narrative that they identify.

In a similar vein, yet independent in value, metrics and KPIs create the ability to quantify achievement and impact in unparalleled ways. In their truest form, the beauty is that they can’t be gamed. You either hit them or you don’t; you reap the rewards or incur the penalty. Other communicated factors for success are softer; manipulated and massaged messages telling the story that you’d like or the one others would hope you to believe. While these messages are cloudy and limit transparency, well defined metrics can become your single points of truth and KPIs can help to raise your accountability both internally and externally.

To consistently deliver value and authentic results, designs of metrics should always be distilled down to data which materially impacts the process at hand while striving to avoid unnecessary information. Metrics benchmark your success, and infer what improvements can be made in future sprints. In my opinion however, KPIs are more pronounced, and at their best signify strategic objectives and goals leading to your overall success. Given the growth of both small and large technology integrators it has become imperative for me to ensure that KPIs (often contractually), orient teams to a single mission. Unfortunately, there have been times this wasn’t the case, and we have felt the pain of spin and discord. However, when designed correctly, and supported by tactical metrics, the results have helped ensure successful delivery of many of our initiatives. In looking forward, my hope for the generation of technologists on the horizon is that they leverage the power and influence of analytics, metrics, and KPIs and use them to quantify their value and impact to their organizations.

Grant Small, Nike Supply Chain Technology Consultant

Google Analytics is a robust analytics engine developed to help websites maximize advertising return on investment (ROI) and track detailed behavioral metrics. The tool itself is easy to use; just set up an account, copy a few lines of Javascript code to inject into your own product, and the results will autopopulate in the dashboard once user interaction begins.

In Figure 8-4, the Google Analytics demo instance is up and running, giving us a peek into the main dashboard view. A grouping of high-level metrics is shown (Users, Revenue, Conversion rate, Sessions) along with sections to go deeper in the left-hand navigation pane.

Google Analytics splits the reports into four sections: Audience, Acquisition, Behavior, and Conversions. Since the tool collects so much information at once, the subsections are helpful when honing in on related slices of the analytical data set.

For a product manager, the behavior flow is a critical view to develop familiarity with (Figure 8-5). It tracks the number of users who touch the landing page, and shows a raw user count for each set of possible interactions that users could follow in the product. This report can identify areas with high rates of exit, and confirm if the product is intuitive enough to guide users from start to finish without hand-holding.

Another helpful view to keep an eye on is the funnel visualization (Figure 8-6). The funnel begins with a top-level action or function (in this case, the shopping cart) and tracks a user’s journey through the flow to show how many end up past billing & shipping, payment, and finally to checkout. The funnel can expose bottlenecks in the system as users are closing in on the main goal, and elements that prevent them from a successful positive exit scenario.

Experiment with the Google Analytics dashboard and make a note of the sections of data that suit a particular product’s industry, type, technology stack, and use case. Then, create a set of reports that return only the set of numbers you need to make informed decisions. Everything else can be treated as random noise.

Collect everything you can, then trim it down. The biggest mistake you can make is to assume you only need X amount of data points, then realize later down the road that X + Y is what you REALLY needed. Even if you have to find the needle in the haystack later on by storing so much data, it’s worth it compared to making up for lost time and starting the collection today as opposed to yesterday.

Collect, track, measure, and influence. You will be at a grave disadvantage if you don’t.