As you evaluate your company’s strategy, perhaps even considering a strategic pivot, you’ll want to gather and utilize all available data to build a deep understanding of your customers, your competitors, the external factors that impact you and even your own product. The big data ecosystem will play an important role in this process, enabling insights and guiding actions in ways not previously possible.
Your customers
Customer data is one of your most important assets. There is more data available today than ever before, and it can tell you much more than you’d expect about your customers and potential customers: who they are, what motivates them, what they prefer and what their habits are. The more data you collect, the more complete your customer picture will become, so make it your goal to collect as much data as possible, from as many sources as possible.
Getting the data
Start by identifying all customer interaction points:
- Visits to your digital platforms: websites, apps and kiosks’.
- Interactions with customer support: phone, email, online chat, etc.
- Social media, including direct messaging, tweets, and posts on accounts you own or they own’.
- Records of physical movement, including store videos and movement logs. There are several technologies for monitoring movement, including embedded sensors, Wi-Fi, Bluetooth, beacons, and even light frequencies in combination with smartphone apps.
- In some industries, you will have access to additional (non-geo) sensor data from sensors, RFID tags, personal fitness trackers, etc., which may provide data such as bio-medical readings, accelerometer data, external temperature, etc.
For each interaction point, make an inventory of:
- what data you can possibly collect;
- what the potential uses of that data are; and
- what privacy and governance factors you should consider (see Chapter 11).
For many organizations, the interaction points will consist of physical and web stores, Apple (iOS) and Android apps, social media channels, and staff servicing customers in stores, call centres, online chat and social media. I’ll illustrate with a few examples.
Digital
Start with your digital platforms. First the basics (not big data yet).
You’ll probably already have some web analytics tags on your website that record high-level events. Make sure you also record the key moments in the customer journey, such as when the visitor does an onsite search, selects filters, visits specific pages, downloads material, watches your videos or places items in the checkout basket. Record mouse events, such as scrolls and hovers. Make sure these moments are tagged in a way that preserves the details you’ll need later, such as adding the details of product category, price range, and product ID to the web tags associated with each item description page. This will allow you to quickly do top-of-mind analysis, such as identifying how often products in certain categories were viewed, or how effective a marketing campaign was in driving a desired event. In the end, you’ll probably have several dozen or even several hundred specific dimensions that you add to your out-of-the-box web analytics data. This isn’t yet big data.
With the additional detailed tags that you’ve implemented, you’ll be able to analyse and understand many aspects of the customer journey, giving insights into how different types of customers interact with the products you’ve presented them. We’ll show examples of this below.
If you haven’t already done so, set up conversion funnels for sequential events that lead to important conversion events, such as purchases. Figure 6.1 shows how a basic purchase funnel might look.
Each intermediate goal in the conversion funnel is a micro-conversion, together leading to a macro-conversion (‘checkout’ in this case). Choose your micro-conversions in a way that reflects increasing engagement and increased likelihood of a final conversion. The funnels you set up will enable you to analyse drop-off rates at each stage, allowing you to address potential problem points and increase the percentage of visitors progressing along each stage of the funnel, eventually reaching the conversion event at the end of the funnel. Depending on your product, customer movement down the funnel may span several days, weeks or months, so you’ll need to decide what to consider ‘drop-off’.
For privacy and governance in your website, research and comply with local laws governing the use of web cookies. Make a list of where you are storing the browsing data that identifies the individual user (such as IP address) and how you later use the insights you’ve gathered in customizing your interactions with each user. For example, if you personalize your content and marketing based on the users’ online actions, you’ll need to consider the ethical and legal implications. Remember the example from Target.
Now the big data part. You’ve set up the web analytics to record details most meaningful for you. Now hook your web page up to a big data system that will record every online event for every web visitor. You’ll need a big data storage system, such as HDFS, and you’ll need to implement code (typically JavaScript) that sends the events to that storage. If you want a minimum-pain solution, use Google Analytics’ premium service (GA360), and activate the BigQuery integration. This will send your web data to Google’s cloud storage, allowing you to analyse it in detail within a few hours. If you need data in real time, you can change the GA JavaScript method sendHitTask and send the same data to both Google and to your own storage system. Such an architecture is illustrated below in Figure 6.2. Note that Google’s terms and conditions require that you not send personally identifiable information (PII) (we’ll discuss PII further in Chapter 11).
Figure 6.2 Example architecture for a streaming big data implementation.
Source: icons from BigQuery, Apache Kafka, Jupyter notebooks and Tableau.
You’ll now have the raw customer (big) data you need to formulate a very detailed understanding of your customers, as described later in this chapter.
Customer support
Consider recording and analysing all interactions with sales agents and customer support: phone calls, online chats, emails and even videos of customers in stores. Most of this data is easy to review in pieces, but difficult to analyse at scale without advanced tools. As you store these interactions, your customer support agents should enrich them with additional information, such as customer ID and time of day’, and label them with meaningful categories, such as ‘order enquiry’, ‘new purchase’, ‘cancellation’ or ‘complaint.’ You can then save the entire data file in a big data storage system (such as MongoDB or HDFS). We’ll show valuable ways to use this data later in this chapter.
Physical movements
You have a choice of several technologies for monitoring how your customers are moving within your stores. In addition to traditional video cameras and break-beam lasers across entrances, there are technologies that track the movement of smartphones based on cellular, Bluetooth or Wi-Fi interactions. Specialized firms such as ShopperTrak and Walkbase work in these areas. Such monitoring will help you understand the browsing patterns of your customers, such as what categories are considered by the same customers and how much time is spent before a purchase decision. It will help you direct your register and support staff where needed. Again, this data is valuable even if the customer is kept anonymous.
When a customer arrives at the register and makes a purchase, possibly with a card that is linked to that customer, you will be able to see not only what is being purchased, but also what other areas of the store were browsed. You might use this information in future marketing or you might use it to redesign your store layout if you realize that the current layout is hampering cross-sell opportunities.
These are just a few examples. In general, start collecting and storing as much detail as possible, making sure to consider business value, to respect customer privacy and to comply with local laws in your collection, storage and use of this data. Be careful not to cross the line between ‘helpful’ and ‘creepy’. Keep your customers’ best interests in mind and assume any techniques you use will become public knowledge.
Keep in mind
Try to provide a useful service to your customer for each piece of privacy you ask them to sacrifice. For example, if your smartphone app tracks a customer’s physical location, make sure the customer gets valuable location-based services from this app. Also, provide a better, personalized online experience to visitors who are logged in to your website. In this way, your interests are aligned with those of your customers.
Linking customer data
Link the customer data from your interaction points to give a holistic picture of customer journey. If a customer phones your call centre after looking online at your cancellation policy webpage, you should be able to connect those two events in your system. To do this, enter a unique customer field (such as phone number or user name) along with the call record.
If you are monitoring a customer walking through your store, link that footpath with subsequent register sales information (subject to privacy considerations). Do this by recording the timestamp and location of the point of sale with the footpath data. Combine the data centrally to give the complete picture.
Sometimes you’ll use anonymous customer data, such as when analysing traffic flow. Other times you’ll use named-customer data, such as when analysing lifetime activity. For the named-customer applications, you’ll want to de-duplicate customers. This is difficult, and you’ll probably have limited success. The best situation is when customers always present a unique customer ID when using your service. In an online setting, this would require a highly persistent and unique login (as with Facebook). Offline, it typically requires photo ID. In most situations, you won’t have this luxury, so use best efforts to link customer interactions.
You’ll typically face the following problems in identifying your customers:
- Problem: Customers will not identify themselves (e.g. not logging in).
- Possible solutions: Use web cookies and IP addresses to link visits from the same visitors, producing a holistic picture of anonymous customer journeys extended across sessions. Use payment details to link purchases to customers. Smartphones may provide information to installed apps that allow additional linking. Talk to your app developers about this.
- Problem: Customers create multiple logins.
- Possible solutions: Clean your customer database by finding accounts that share key fields: name, email address, home address, date of birth, or IP address. A graph database such as Neo4J can help in this process, as illustrated in Figure 6.3. Work with the business to create logic for which customers to merge and which to associate using a special database field (e.g. ‘spouse of’). Change your account creation process to detect and circumvent creation of duplicate accounts, such as by flagging email addresses from existing accounts.
Figure 6.3 A graph database can help de-duplicate customers.
Using the data
Much of your customer data will be useful even in the aggregate and anonymized format provided by your standard web analytics tool. You’ll see how many customers came at any hour of the day as well as useful information such as average time spent onsite, number of pages viewed, how many entered at each page or from each marketing campaign, etc. You’ll also see the total transactions made by customer segments, such as geography and acquisition source. This will give you a picture of how and when your products are being used, particularly when matched against marketing campaigns, holidays, service downtimes and new initiatives.
Customer journey
The big data insights get much more useful when you use data to build an understanding of the intents, preferences and habits of your customers. You should already be segmenting your customers into personas based on static features such as home address, gender, language, age and possibly income level. Note that Google Analytics can provide useful demographic information (from their DoubleClick cookie) if you enable this functionality.
Broaden your segmentation criteria to include customer journey data, such as those listed below.
- What filters do they most often use? Is it Price high-to-low? Price low-to-high? Highest rated item? Newest product? Customers sorting price low-to-high are probably price conscious. Those sorting price high-to-low or by highest rating are probably quality conscious. Those sorting newest first may be quality conscious or perhaps technophiles or early adopters. Those sorting by rating may be late adopters or quality conscious. All of this will impact how you interact with them. If a customer is quality conscious but not price conscious, you should present them with high-quality products in search results and marketing emails, and you should not be presenting them with clearance sales of entry-level products. You’ll want to interact with the price-conscious customer segments in exactly the opposite way.
- How many items do they typically consider before making a purchase? This information will help you decide when to intervene in the shopping process, such as by offering a discount when a customer is about to leave without making a purchase.
- What categories do they visit most often? This will help you segment the customer and return the most relevant results for ambiguous search phrases (such as ‘jaguar’ the car vs ‘jaguar’ the animal or Panama City, Florida vs Panama City, Panama). You’ll also use this information to guide which items you market to the customer.
- Do they change the shipping option to save fees? Again, a signal of price-conscious customers.
- Do they read customer reviews? Do they write reviews? If they always read reviews, don’t present them with poorly reviewed items in your marketing emails, search results, or cross-sell frames. If they often write reviews, or if they own social media accounts with an unusually high number of followers, make sure they get ‘golden glove’ customer support.
- What types of marketing do they best respond to? Do they open newsletters? Do they respond to flash sales? Don’t fill their inbox with marketing they never respond to. Give them the most relevant media and you’ll increase the odds they respond rather than clicking ‘unsubscribe’.
- If they are active on social media, what are the topics and hashtags they most frequently mention? Can you use this knowledge to market more effectively to them? Here again, you are building customer personas with all relevant information. Try to get connected to their social media accounts. As we mentioned in Chapter 4, what they do here can tell you a lot about them.
Customer segments (personas)
Using data you’ve collected, decide what factors are most meaningful in dividing your customers into segments (personas). Examples would be ‘price-sensitive males aged 20 to 30’ or perhaps ‘high-spending technophiles who make quick purchasing decisions’, or ‘customers who browse daily across specific categories but buy only on discount’. You can construct these segments in a qualitative way, using the intuition of marketing experts guided by the data, or you can construct the segments in a quantitative way, using analytic tools such as clustering and principal component analysis. Both are valid methods, but if you have a lot of data that can be segmented in many ways, the quantitative approach will probably be more effective.
Inventory
This customer journey data will give deeper insights into how your inventory impacts customer experience. As you consider removing items with low sales, the customer journey data may show a high percentage of profitable transactions by customers who found your site searching for those non-profitable products. You’ll want to understand more about the customer journey of those customers, what they looked for and why they made their decisions, before you make the decision to remove products which may have attracted those customers to your shop in the first place.
On the other hand, you may be selling a product that is often showing up in online search results but is of no interest to your customers, as demonstrated by your customer journey data. In this case, you should either change the display of the product search result, or remove the item altogether, as it is taking up valuable search real estate.
Critical intervention
Apply both basic analysis and advanced machine learning to your customer data and you’ll likely find ways to decrease churn and increase sales. Basic analysis of where and when your customers are active will help you with scheduling the shifts and skill sets of your support personnel. It will also signal what the customer is likely to do next (a customer visiting your appliance store twice last week may be preparing for a significant purchase).
With a bit more analysis, you’ll start detecting subtle but important signals. A European telecommunications company recently analysed data on customers cancelling their subscriptions and found that a large number followed the same three or four steps prior to cancellation, such as reviewing their contract online, then phoning customer support, then disputing a bill in person and then cancelling the contract. By linking those events, the company identified signals of impending churn so it could take action.
At an even more advanced level, machine learning techniques could detect pending account cancellation or the likelihood of a sale based on an analysis of text, audio or video. Such a system might be a significant investment of your time and resources, but you might have the business case to justify it, or you might find a vendor who has already developed a technology suitable for your application, as illustrated in the next case study.
Case study – Actionable customer insights from in-store video feeds
The Economist magazine recently wrote about two innovations in the use of video analysis. Realeyes, an emotion-detection firm based in London, found that shoppers who entered a store smiling spent a third more than others. In another pilot programme, a European bookstore chain began using software from Angus.ai to monitor when its customers walked to the end of an aisle and returned with a frown. The software then discretely messaged a sales clerk to help. The result was a 10 per cent rise in sales.53
As always, consult your company’s privacy officer to stay within the limits of the law, stay aligned with your customers’ best interests and don’t do anything that would give negative publicity if it were made public.
Your competitors
It’s especially challenging to get good information about your competitors. Information brokers such as Nielsen, Comscore and SimilarWeb will sell their estimations of traffic to your competitors’ sites and apps, possibly including referrer information. The website trends.google.com gives charts for the number of searches for specific terms, which in turn give indications of how you compare with your competitors for brand search (see Figure 6.4).
Figure 6.4 Searches in Brazil for ‘McDonalds’ (top line) vs ‘Burger King’ (bottom line) Q2, 2017 (Google Trends).
You’ll be able to get information on competitor inventory, services and physical locations by scraping websites. Your technology team can help with this (you can use a tool such as Selenium). If you are competing on price, you’ll want to adjust your pricing based on your competition’s proximity to your customers. For physical locations, that will be based on address and transportation routes. For online sales, that will be partially influenced by the referrer sending a visitor to your site. Customers arriving from price comparison sites should be considered price-conscious and at high risk of buying from your competitors.
Work to increase your share of wallet, the percentage of a customer’s spend that goes to your business rather than to competitors’. Start by using the detailed customer data you have collected and see what categories and what specific products are typically purchased by the same customer segments. You’ll be able to see which customers are already making those cross-purchases, which are browsing but not purchasing, and which are active in only one category.
Identify the products your customers are purchasing elsewhere to see where you are losing share of wallet. If you sell groceries and your customer only buys fruits and vegetables, you’ll know they are buying milk and eggs elsewhere. If you sell electronics and they only buy smartphones, you’ll know they are buying computers elsewhere. This will help identify areas where you need to compete harder. By using the customer segments you’ve created, you’ll see if your competition is appealing more to quality-conscious customers, marketing-reactive customers, high-spenders, etc.
Monitor online job boards to get insights into competitors’ initiatives. A significant increase in postings for a location or job function will indicate activity in that area. Create watch lists of competitor employees’ LinkedIn profiles and monitor them for anomalies in profile updates. If an unusually high number of employees are updating their LinkedIn profiles, it may signal turmoil or pending layoffs within the company. Similarly, natural language processing run on company public statements can uncover unusual activity. This technique has been used effectively to signal pending initial public offerings.
External factors
Your strategy will be influenced by factors ranging from government regulation to local weather. If you are in the travel and tourism industry, regional holidays will impact long-range bookings and weather will influence impulse bookings. The price of commodities will influence production and transport costs, and exchange rates or political turmoil will influence cross-border activity.
Much of the impact of external factors will be from traditional (small) data, but newer (big) data sources will provide additional, valuable signals. Keep an eye on changes in online customer activity, which may signal unexpected factors requiring your attention. To illustrate, consider how Google Maps and Waze can detect construction or road closures simply by studying driver movements.
To give another example, you may not be aware of the release of an innovative new product until you see it in onsite searches or detect the impact in sales of your other products. If you are running a hotel chain and have a property in Scranton, Pennsylvania, you may have no idea there is a major convention about an obscure topic being planned there during the second week in February. If you are prepared with a forecast of booking rates for February, you’ll see the unexpected spike in the customer activity in your booking site and call centres already in October, before you even know about the February conference. By monitoring customer activity, you can act to raise room rates in October before running out of underpriced February inventory a few weeks later.
To this end, you should construct regular forecasts of key metrics, including number of visits and sales projections. You’ll do this by consulting historic figures, projecting growth rates, and speaking with your business units to consider anything out of the ordinary (holiday cycles, major events, regulatory or economic changes, etc.). These forecasts should be segmented down to the levels at which you can steer your operations (such as product and region) and should preferably be made at daily granularity. If you automatically monitor these figures at daily or weekly granularities you can raise an alert whenever they move above or below expected levels, signalling when some external factor is impacting your business in an unexpected way.
Your own product
You need to truly understand your own service and product offerings when evaluating your strategy. You may not understand them as well as you think, and your customers may perceive them in completely different ways than you’d expect. What is working and what is not working? How are customers responding to your products? Where are you losing money through inefficiencies?
If your web offering is a significant part of your business, find out what is working there and work to make it better. Create and track micro-conversions to see how your items are performing even before a purchase is made. These provide valuable insights even if they are not part of a funnel analysis.
Track the customer engagement with your other digital offerings.
- What is the applause rate on your social media? (How many times are your tweets liked or retweeted? How many comments on your Facebook posts?)
- How many people download your material?
- How many people sign up for your newsletter?
Test changes in your products by running A/B tests, which you’ll do in the following way:
| 1. | Propose one small change that you think may improve your offering. Change one frame, one phrase, or one banner. Check with your development team to make sure it’s an easy change. | |
| 2. | Decide what key performance indicators (KPI) you most want to increase: revenue, purchases, up-sells, time onsite, etc. Monitor the impact on other KPIs. | |
| 3. | Run the original and the changed version (A and B) simultaneously. For websites, use an A/B tool such as Optimizely. View the results using the tool or place the test version ID in web tags and analyse specifics of each version, such as by comparing lengths of path to conversion. | |
| 4. | Check if results are statistically significant using a two-sample hypothesis test. Have an analyst do this or use an on-line calculator such as https://abtestguide.com/calc/. | |
| 5. | Use your big data system for deeper analysis: | |
| a. | Were there significant changes in customer journey, such as number of categories viewed or filters selected? | |
| b. | Are there key product or customer segments you should manage differently? | |
| c. | Did specific external events influence results? | |
| d. | Did KPIs move in different directions? | |
Align your assumptions about your product with these new insights. For example:
- Are you trying to compete on price, while most of your revenue is coming from customers who are quality conscious?
- Are you not taking time to curate customer reviews, while most of your customers are actively sorting on and reading those reviews?
If your assumptions about your product don’t align with what you learn about your customers’ preferences and habits, it may be time for a strategic pivot.
Use modern data and data science (analytics) to get the insights you’ll need to determine and refine your strategy. Selectively choose the areas in which you should focus your efforts in (big) data and data science and then determine the necessary tools, teams and processes.
In the next chapter, I’ll talk about how to choose and prioritize your data efforts.
Takeaways
- Big data sources will help inform your strategy by giving new insights into your customers, competition, business environment and product.
- There are many new sources of non-traditional data. Take an inventory of what is available and what is most useful.
- You will typically have difficulty linking your customer actions across different touch points.
- Your website and other digital portals can provide detailed information about customer intentions, preferences and habits, signalling when you need to make a tactical change or a strategic pivot.
- Running your A/B tests in combination with a big data system allows you to gather much deeper insights.
Ask yourself
- Make a list of your customer touchpoints. For each, note if you are digitalizing and storing none, some, or all the available data. Rate each touchpoint from 1 to 10 in terms of (a) the value of the data and (b) how difficult it would be to store and analyse additional data from that touchpoint. Multiply those two scores together. The touchpoints with the highest resulting numbers are your most promising new sources of customer data.
- What data sources would you need to link to get a full view of your customer interactions? Is there anything preventing you from linking this data?
- What are ways in which your customers differ in their preferences and behaviour and which might impact the way in which you relate to them and the products and experiences you offer to them?
- What have been the most successful product changes that you made after testing the results of different possibilities?
- Which external data is relevant for your organization: economic data, weather, holiday schedules, etc.?
- Which data sources could give you more insight into your competition? Think of private and public information providers as well as graphs and signals provided by internet companies such as Google and LinkedIn.