Background

Amazon long ago outgrew their original business model of an online bookshop. They are now one of the world’s largest retailers of physical goods, virtual goods such as ebooks and streaming video and more recently Web services.

Much of this has been built on top of their pioneering use of “recommendation engine” technology – systems designed to predict what we want, when we want it and of course offer us the chance to give them money for it.

With this ethos in mind, Amazon have also moved into being a producer of goods and services, rather than just a retailer. As well as commissioning films and TV shows, they build and market electronics, including tablets, TV boxes and streaming hardware.

Even more recently, they have moved to take on food supermarkets head-on by offering fresh produce and far quicker delivery through their Amazon Now service.

What Problem Is Big Data Helping To Solve?

Information overload is a very real problem, and retailers have more to lose from it than most of us. Online retailing relies on making as large a number of products or services available as possible, to increase the probability of making sales. Companies like Amazon and Walmart have thrived by adopting an “everything under one roof” supermarket model.

The problem here is that a customer can often feel overwhelmed when presented with a huge range of possible options. Psychologically, worries about suffering from “buyer’s remorse” – wasting money by making ill-informed purchasing decisions – can lead to our putting off spending money until we are certain we have done sufficient research. The confusing amount of options may even cause us to change our minds entirely about the fact we need a $2,000 ultraHD television set and decide to go on vacation instead.

It’s the same problem that often plagues many projects involving large amounts of information. Customers can become data-rich (with a great many options) but insight-poor – with little idea about what would be the best purchasing decision to meet their needs and desires.

How Is Big Data Used In Practice?

Essentially, Amazon have used Big Data gathered from customers while they browse the site to build and fine-tune their recommendation engine.

Amazon probably didn’t invent the recommendation engine but they introduced it to widespread public use. The theory is that the more they know about you, the more likely they are to be able to predict what you want to buy. Once they’ve done that, they can streamline the process of persuading you to buy it by cutting out the need for you to search through their catalogue.

Amazon’s recommendation engine is based on collaborative filtering. This means that it decides what it thinks you want by working out who you are, then offering you items that people with a similar profile to you have purchased.

Unlike with content-based filtering – as seen, for example, in Netflix’s recommendation engine – this means the system does not actually have to know anything about the unstructured data within the products it sells. All it needs is the metadata: the name of the product, how much it costs, who else has bought it and similar information.

Amazon gather data on every one of their over a quarter of a billion customers while they use their services.¹ As well as what you buy, they monitor what you look at, your shipping address to determine demographic data (they can take a good stab at guessing your income level by knowing what neighbourhood you live in) and whether you leave customer reviews and feedback.

They also look at the time of day you are browsing, to determine your habitual behaviours and match your data with others who follow similar patterns.

If you use their streaming services, such as Amazon Prime streaming video or ebook rental, they can also tell how much of your time you devote to watching movies or reading books.

All of this data is used to build up a “360-degree view” of you as an individual customer. Based on this, Amazon can find other people who they think fit into the same precisely refined consumer niche (employed males between 18 and 45, living in a rented house with an income of over $30,000 who enjoy foreign films, for example) and make recommendations based on what they like.

In 2013, Amazon began selling this data to advertisers, to allow them to launch their own Big Data-driven marketing campaigns. This put them in competition with Google and Facebook, which also sell anonymized access to user data to advertisers.

What Were The Results?

Amazon have grown to become the largest online retailer in the US based on their customer-focused approach to recommendation technology. Last year, they took in nearly $90 billion from worldwide sales.

Revenues for their cloud-based Web services businesses such as Amazon Web Services have grown 81% in the last year, to $1.8 billion.

In addition, Amazon’s approach to Big Data-driven shopping and customer services has made them a globally recognized brand.

What Data Was Used?

Amazon collect data from users as they browse the site – monitoring everything from the time they spend browsing each page to the language used in the user reviews they leave. Additionally, they use external datasets such as census information to establish our demographic details. If you use their mobile apps on your GPS-enabled smartphone or tablet, they can also gather your location data and information about other apps and services you use on your phone. Using Amazon’s streaming content services, such as Amazon Prime and Audible, provides them with more detailed information on where, when and how you watch and listen to TV, film and audio.

What Are The Technical Details?

Amazon’s core business is handled in their central data warehouse, which consists of Hewlett-Packard servers running Oracle on Linux, to handle their 187 million unique monthly website visitors, and over two million third-party Amazon Marketplace sellers.

Any Challenges That Had To Be Overcome?

In the early days, by far the biggest challenge for Amazon and all e-tailers was getting the public to put their faith in taking part in online commercial activity. These days, thanks to enhanced security and legislative pressure (and in spite of ever-increasing incidences of data theft), most of us are no more wary of giving our card details to an online retailer than we are to a bricks ‘n’ mortar one. Amazon use Netscape Secure Commerce Server systems and SSL to store sensitive information in an encrypted database.

What Are The Key Learning Points And Takeaways?

Diversity of consumer choice is a great thing, but too much choice and too little guidance can confuse customers and put them off making purchasing decisions.

Big Data recommendation engines simplify the task of predicting what a customer wants, by profiling them and looking at the purchase history of people who fit into similar niches.

The more a business knows about a customer, the better it can sell to them. Developing a 360-degree view of each customer as an individual is the foundation of Big Data-driven marketing and customer service.

Privacy and information security is an absolute priority. One large-scale data breach or theft can destroy consumer confidence in a business overnight.

REFERENCES AND FURTHER READING

1. Statista (2015) Annual number of worldwide active Amazon customer accounts from 1997 to 2014 (in millions), http://www.statista.com/statistics/237810/number-of-active-amazon-customer-accounts-world-wide/, accessed 5 January 2016.

For further details on Amazon’s use of Big Data, see:

1. https://datafloq.com/read/amazon-leveraging-big-data/517
2. http://expandedramblings.com/index.php/amazon-statistics/
3. Scientific explanation of Amazon’s recommendation engine, and collaborative filtering in general, by Greg Linden, Brent Smith, and Jeremy York of Amazon. http://www.scribd.com/doc/14771615/Amazon-Recommendations