As you evaluate your company’s strategy, perhaps even considering a strategic pivot, you’ll want to gather and utilize all available data to build a deep understanding of your customers, your competitors, the external factors that impact you and even your own product. The big data ecosystem will play an important role in this process, enabling insights and guiding actions in ways not previously possible.
Customer data is one of your most important assets. There is more data available today than ever before, and it can tell you much more than you’d expect about your customers and potential customers: who they are, what motivates them, what they prefer and what their habits are. The more data you collect, the more complete your customer picture will become, so make it your goal to collect as much data as possible, from as many sources as possible.
Start by identifying all customer interaction points:
For each interaction point, make an inventory of:
For many organizations, the interaction points will consist of physical and web stores, Apple (iOS) and Android apps, social media channels, and staff servicing customers in stores, call centres, online chat and social media. I’ll illustrate with a few examples.
Start with your digital platforms. First the basics (not big data yet).
You’ll probably already have some web analytics tags on your website that record high-level events. Make sure you also record the key moments in the customer journey, such as when the visitor does an onsite search, selects filters, visits specific pages, downloads material, watches your videos or places items in the checkout basket. Record mouse events, such as scrolls and hovers. Make sure these moments are tagged in a way that preserves the details you’ll need later, such as adding the details of product category, price range, and product ID to the web tags associated with each item description page. This will allow you to quickly do top-of-mind analysis, such as identifying how often products in certain categories were viewed, or how effective a marketing campaign was in driving a desired event. In the end, you’ll probably have several dozen or even several hundred specific dimensions that you add to your out-of-the-box web analytics data. This isn’t yet big data.
With the additional detailed tags that you’ve implemented, you’ll be able to analyse and understand many aspects of the customer journey, giving insights into how different types of customers interact with the products you’ve presented them. We’ll show examples of this below.
If you haven’t already done so, set up conversion funnels for sequential events that lead to important conversion events, such as purchases. Figure 6.1 shows how a basic purchase funnel might look.
Each intermediate goal in the conversion funnel is a micro-conversion, together leading to a macro-conversion (‘checkout’ in this case). Choose your micro-conversions in a way that reflects increasing engagement and increased likelihood of a final conversion. The funnels you set up will enable you to analyse drop-off rates at each stage, allowing you to address potential problem points and increase the percentage of visitors progressing along each stage of the funnel, eventually reaching the conversion event at the end of the funnel. Depending on your product, customer movement down the funnel may span several days, weeks or months, so you’ll need to decide what to consider ‘drop-off’.
For privacy and governance in your website, research and comply with local laws governing the use of web cookies. Make a list of where you are storing the browsing data that identifies the individual user (such as IP address) and how you later use the insights you’ve gathered in customizing your interactions with each user. For example, if you personalize your content and marketing based on the users’ online actions, you’ll need to consider the ethical and legal implications. Remember the example from Target.
Now the big data part. You’ve set up the web analytics to record details most meaningful for you. Now hook your web page up to a big data system that will record every online event for every web visitor. You’ll need a big data storage system, such as HDFS, and you’ll need to implement code (typically JavaScript) that sends the events to that storage. If you want a minimum-pain solution, use Google Analytics’ premium service (GA360), and activate the BigQuery integration. This will send your web data to Google’s cloud storage, allowing you to analyse it in detail within a few hours. If you need data in real time, you can change the GA JavaScript method sendHitTask and send the same data to both Google and to your own storage system. Such an architecture is illustrated below in Figure 6.2. Note that Google’s terms and conditions require that you not send personally identifiable information (PII) (we’ll discuss PII further in Chapter 11).
Figure 6.2 Example architecture for a streaming big data implementation.
Source: icons from BigQuery, Apache Kafka, Jupyter notebooks and Tableau.
You’ll now have the raw customer (big) data you need to formulate a very detailed understanding of your customers, as described later in this chapter.
Consider recording and analysing all interactions with sales agents and customer support: phone calls, online chats, emails and even videos of customers in stores. Most of this data is easy to review in pieces, but difficult to analyse at scale without advanced tools. As you store these interactions, your customer support agents should enrich them with additional information, such as customer ID and time of day’, and label them with meaningful categories, such as ‘order enquiry’, ‘new purchase’, ‘cancellation’ or ‘complaint.’ You can then save the entire data file in a big data storage system (such as MongoDB or HDFS). We’ll show valuable ways to use this data later in this chapter.
You have a choice of several technologies for monitoring how your customers are moving within your stores. In addition to traditional video cameras and break-beam lasers across entrances, there are technologies that track the movement of smartphones based on cellular, Bluetooth or Wi-Fi interactions. Specialized firms such as ShopperTrak and Walkbase work in these areas. Such monitoring will help you understand the browsing patterns of your customers, such as what categories are considered by the same customers and how much time is spent before a purchase decision. It will help you direct your register and support staff where needed. Again, this data is valuable even if the customer is kept anonymous.
When a customer arrives at the register and makes a purchase, possibly with a card that is linked to that customer, you will be able to see not only what is being purchased, but also what other areas of the store were browsed. You might use this information in future marketing or you might use it to redesign your store layout if you realize that the current layout is hampering cross-sell opportunities.
These are just a few examples. In general, start collecting and storing as much detail as possible, making sure to consider business value, to respect customer privacy and to comply with local laws in your collection, storage and use of this data. Be careful not to cross the line between ‘helpful’ and ‘creepy’. Keep your customers’ best interests in mind and assume any techniques you use will become public knowledge.
Try to provide a useful service to your customer for each piece of privacy you ask them to sacrifice. For example, if your smartphone app tracks a customer’s physical location, make sure the customer gets valuable location-based services from this app. Also, provide a better, personalized online experience to visitors who are logged in to your website. In this way, your interests are aligned with those of your customers.
Link the customer data from your interaction points to give a holistic picture of customer journey. If a customer phones your call centre after looking online at your cancellation policy webpage, you should be able to connect those two events in your system. To do this, enter a unique customer field (such as phone number or user name) along with the call record.
If you are monitoring a customer walking through your store, link that footpath with subsequent register sales information (subject to privacy considerations). Do this by recording the timestamp and location of the point of sale with the footpath data. Combine the data centrally to give the complete picture.
Sometimes you’ll use anonymous customer data, such as when analysing traffic flow. Other times you’ll use named-customer data, such as when analysing lifetime activity. For the named-customer applications, you’ll want to de-duplicate customers. This is difficult, and you’ll probably have limited success. The best situation is when customers always present a unique customer ID when using your service. In an online setting, this would require a highly persistent and unique login (as with Facebook). Offline, it typically requires photo ID. In most situations, you won’t have this luxury, so use best efforts to link customer interactions.
You’ll typically face the following problems in identifying your customers:
Figure 6.3 A graph database can help de-duplicate customers.
Much of your customer data will be useful even in the aggregate and anonymized format provided by your standard web analytics tool. You’ll see how many customers came at any hour of the day as well as useful information such as average time spent onsite, number of pages viewed, how many entered at each page or from each marketing campaign, etc. You’ll also see the total transactions made by customer segments, such as geography and acquisition source. This will give you a picture of how and when your products are being used, particularly when matched against marketing campaigns, holidays, service downtimes and new initiatives.
The big data insights get much more useful when you use data to build an understanding of the intents, preferences and habits of your customers. You should already be segmenting your customers into personas based on static features such as home address, gender, language, age and possibly income level. Note that Google Analytics can provide useful demographic information (from their DoubleClick cookie) if you enable this functionality.
Broaden your segmentation criteria to include customer journey data, such as those listed below.
Using data you’ve collected, decide what factors are most meaningful in dividing your customers into segments (personas). Examples would be ‘price-sensitive males aged 20 to 30’ or perhaps ‘high-spending technophiles who make quick purchasing decisions’, or ‘customers who browse daily across specific categories but buy only on discount’. You can construct these segments in a qualitative way, using the intuition of marketing experts guided by the data, or you can construct the segments in a quantitative way, using analytic tools such as clustering and principal component analysis. Both are valid methods, but if you have a lot of data that can be segmented in many ways, the quantitative approach will probably be more effective.
This customer journey data will give deeper insights into how your inventory impacts customer experience. As you consider removing items with low sales, the customer journey data may show a high percentage of profitable transactions by customers who found your site searching for those non-profitable products. You’ll want to understand more about the customer journey of those customers, what they looked for and why they made their decisions, before you make the decision to remove products which may have attracted those customers to your shop in the first place.
On the other hand, you may be selling a product that is often showing up in online search results but is of no interest to your customers, as demonstrated by your customer journey data. In this case, you should either change the display of the product search result, or remove the item altogether, as it is taking up valuable search real estate.
Apply both basic analysis and advanced machine learning to your customer data and you’ll likely find ways to decrease churn and increase sales. Basic analysis of where and when your customers are active will help you with scheduling the shifts and skill sets of your support personnel. It will also signal what the customer is likely to do next (a customer visiting your appliance store twice last week may be preparing for a significant purchase).
With a bit more analysis, you’ll start detecting subtle but important signals. A European telecommunications company recently analysed data on customers cancelling their subscriptions and found that a large number followed the same three or four steps prior to cancellation, such as reviewing their contract online, then phoning customer support, then disputing a bill in person and then cancelling the contract. By linking those events, the company identified signals of impending churn so it could take action.
At an even more advanced level, machine learning techniques could detect pending account cancellation or the likelihood of a sale based on an analysis of text, audio or video. Such a system might be a significant investment of your time and resources, but you might have the business case to justify it, or you might find a vendor who has already developed a technology suitable for your application, as illustrated in the next case study.
The Economist magazine recently wrote about two innovations in the use of video analysis. Realeyes, an emotion-detection firm based in London, found that shoppers who entered a store smiling spent a third more than others. In another pilot programme, a European bookstore chain began using software from Angus.ai to monitor when its customers walked to the end of an aisle and returned with a frown. The software then discretely messaged a sales clerk to help. The result was a 10 per cent rise in sales.53
As always, consult your company’s privacy officer to stay within the limits of the law, stay aligned with your customers’ best interests and don’t do anything that would give negative publicity if it were made public.
It’s especially challenging to get good information about your competitors. Information brokers such as Nielsen, Comscore and SimilarWeb will sell their estimations of traffic to your competitors’ sites and apps, possibly including referrer information. The website trends.google.com gives charts for the number of searches for specific terms, which in turn give indications of how you compare with your competitors for brand search (see Figure 6.4).
Figure 6.4 Searches in Brazil for ‘McDonalds’ (top line) vs ‘Burger King’ (bottom line) Q2, 2017 (Google Trends).
You’ll be able to get information on competitor inventory, services and physical locations by scraping websites. Your technology team can help with this (you can use a tool such as Selenium). If you are competing on price, you’ll want to adjust your pricing based on your competition’s proximity to your customers. For physical locations, that will be based on address and transportation routes. For online sales, that will be partially influenced by the referrer sending a visitor to your site. Customers arriving from price comparison sites should be considered price-conscious and at high risk of buying from your competitors.
Work to increase your share of wallet, the percentage of a customer’s spend that goes to your business rather than to competitors’. Start by using the detailed customer data you have collected and see what categories and what specific products are typically purchased by the same customer segments. You’ll be able to see which customers are already making those cross-purchases, which are browsing but not purchasing, and which are active in only one category.
Identify the products your customers are purchasing elsewhere to see where you are losing share of wallet. If you sell groceries and your customer only buys fruits and vegetables, you’ll know they are buying milk and eggs elsewhere. If you sell electronics and they only buy smartphones, you’ll know they are buying computers elsewhere. This will help identify areas where you need to compete harder. By using the customer segments you’ve created, you’ll see if your competition is appealing more to quality-conscious customers, marketing-reactive customers, high-spenders, etc.
Monitor online job boards to get insights into competitors’ initiatives. A significant increase in postings for a location or job function will indicate activity in that area. Create watch lists of competitor employees’ LinkedIn profiles and monitor them for anomalies in profile updates. If an unusually high number of employees are updating their LinkedIn profiles, it may signal turmoil or pending layoffs within the company. Similarly, natural language processing run on company public statements can uncover unusual activity. This technique has been used effectively to signal pending initial public offerings.
Your strategy will be influenced by factors ranging from government regulation to local weather. If you are in the travel and tourism industry, regional holidays will impact long-range bookings and weather will influence impulse bookings. The price of commodities will influence production and transport costs, and exchange rates or political turmoil will influence cross-border activity.
Much of the impact of external factors will be from traditional (small) data, but newer (big) data sources will provide additional, valuable signals. Keep an eye on changes in online customer activity, which may signal unexpected factors requiring your attention. To illustrate, consider how Google Maps and Waze can detect construction or road closures simply by studying driver movements.
To give another example, you may not be aware of the release of an innovative new product until you see it in onsite searches or detect the impact in sales of your other products. If you are running a hotel chain and have a property in Scranton, Pennsylvania, you may have no idea there is a major convention about an obscure topic being planned there during the second week in February. If you are prepared with a forecast of booking rates for February, you’ll see the unexpected spike in the customer activity in your booking site and call centres already in October, before you even know about the February conference. By monitoring customer activity, you can act to raise room rates in October before running out of underpriced February inventory a few weeks later.
To this end, you should construct regular forecasts of key metrics, including number of visits and sales projections. You’ll do this by consulting historic figures, projecting growth rates, and speaking with your business units to consider anything out of the ordinary (holiday cycles, major events, regulatory or economic changes, etc.). These forecasts should be segmented down to the levels at which you can steer your operations (such as product and region) and should preferably be made at daily granularity. If you automatically monitor these figures at daily or weekly granularities you can raise an alert whenever they move above or below expected levels, signalling when some external factor is impacting your business in an unexpected way.
You need to truly understand your own service and product offerings when evaluating your strategy. You may not understand them as well as you think, and your customers may perceive them in completely different ways than you’d expect. What is working and what is not working? How are customers responding to your products? Where are you losing money through inefficiencies?
If your web offering is a significant part of your business, find out what is working there and work to make it better. Create and track micro-conversions to see how your items are performing even before a purchase is made. These provide valuable insights even if they are not part of a funnel analysis.
Track the customer engagement with your other digital offerings.
Test changes in your products by running A/B tests, which you’ll do in the following way:
1. | Propose one small change that you think may improve your offering. Change one frame, one phrase, or one banner. Check with your development team to make sure it’s an easy change. | |
2. | Decide what key performance indicators (KPI) you most want to increase: revenue, purchases, up-sells, time onsite, etc. Monitor the impact on other KPIs. | |
3. | Run the original and the changed version (A and B) simultaneously. For websites, use an A/B tool such as Optimizely. View the results using the tool or place the test version ID in web tags and analyse specifics of each version, such as by comparing lengths of path to conversion. | |
4. | Check if results are statistically significant using a two-sample hypothesis test. Have an analyst do this or use an on-line calculator such as https://abtestguide.com/calc/. | |
5. | Use your big data system for deeper analysis: | |
a. | Were there significant changes in customer journey, such as number of categories viewed or filters selected? | |
b. | Are there key product or customer segments you should manage differently? | |
c. | Did specific external events influence results? | |
d. | Did KPIs move in different directions? |
Align your assumptions about your product with these new insights. For example:
If your assumptions about your product don’t align with what you learn about your customers’ preferences and habits, it may be time for a strategic pivot.
Use modern data and data science (analytics) to get the insights you’ll need to determine and refine your strategy. Selectively choose the areas in which you should focus your efforts in (big) data and data science and then determine the necessary tools, teams and processes.
In the next chapter, I’ll talk about how to choose and prioritize your data efforts.
3.16.218.221