The objective of this example email campaign is to see whether the buying habits of customers are affected by the type of model used in advertising, with an A/B-style test format. The campaign randomly splits up the creative content delivered in an email campaign. One version of the email shows pictures of models wearing a dress. The other version of the email does not have models; it just shows the same dress on a mannequin.
Examples of the email campaign (in the form of an Excel spreadsheet) can be found in Appendix B at https://github.com/Apress/analytics-for-retail. You need to download this spreadsheet to follow along in the chapter.
Test Goal
The study uses A/B testing to see whether the merchandising in the email impacts click-through, conversion rate, orders, items sold, and revenue, although other factors could affect the outcomes. A/B testing is a form of randomized controlled study, and it is often used in retail marketing. This A/B testing is when there are two types of users assigned randomly, in this case the model and no models, on the landing page to determine the business questions of which method is more successful in attracting buyers to purchase items and increase revenue.
The business questions of this study uses the follows analysis: When using A/B testing, which prototype (model versus no model) affects the click rate, conversion rate, orders purchased, dresses sold, and revenue generated? Also, is there a correlation in the positive direction between conversion rate and revenue for the model and the no model group?
Method
Because these customers are early-morning shoppers, emails were delivered for the month in the early morning at the same time and day of week to both groups as a control. Both groups had the same email subject line and landing page and could not see whether there was a model until they opened the email campaign. Here the campaign is trying to control the variables and show both groups the same content, except for the presence or absence of a model, to determine if models really contribute to customers’ buying habits.
Key performance indicators measured for the campaign include deliverability rate, open rate, click-through rate, conversion rate, and revenue generated. This is achieved via embedded tracking pixels from your CRM system onto the website to attribute on-site behavior and revenue to the campaign.
Since randomized A/B testing was used for both the model and no model image group, the click-through rate to the same website landing page from the email for the average time on-site was controlled.
Data Constants
This type of CRM data can be downloaded to Excel and uploaded from Excel to/from the CRM system, alongside Google Analytics or Core Metrics to run the following statistics.
The dataset for the email campaign presented here was checked, and each group has 30 data points. There were no missing values. There were no outliers or extreme values in the dataset. The minimum, maximum, and range variables for conversions (minimum=5.1, maximum=6, range=0.9), orders (minimum=25, maximum=27, range=2), dresses sold (minimum=30, maximum=41, range=11), and revenue (minimum=3000, maximum=4100, range=1100) for the model, as well as the variables for conversions (minimum=3.6, maximum=6.0, range=2.4), orders (minimum=21, maximum=23, range=2), dresses sold (minimum=22, maximum=30, range=8), and revenue (minimum=2200, maximum=3000, range=800) for the no model group, all looked in the normal scale for each category. The data appeared approximately normally distributed on the histogram (the charts are not shown in the book), and the means and medians were similar for the variables for conversions (mean=5.56, median=5.55), orders (mean=26, median=26), dresses sold (mean=36, median=36), and revenue (mean=3583, median=3550) for the model group as for the variables for conversions (mean=4.8, median=4.7), orders (mean=22, median=22), dresses sold (mean=26, median=26) and revenue (mean=2603, median=2600) for the no model group. The normal curve is a special case where the mean, median, and mode are the same numbers or in real-life approximately the same. (The mode was not calculated because Excel’s mode function does not always work.) Refer to Appendix B for the dataset and the descriptive statistics for the email campaign. At this point, the statistical analysis techniques are ready to be used to describe and analyze the data, the results will be summarized, and the findings will be explained for each business question. The statistical output is in Appendix B for each procedure.
Type of Shopper Targeting
From the store’s database of 300,000 women at a midsize store, the population of loyal shoppers between the ages of 25–35 is 120,000 customers. A previous segmentation study using shopping history derived from loyalty cards had shown that this age group would be most interested in this type of dress and price point from previous buying habits. (Separate campaigns may be sent out to other targeted groups.) This cohort is randomly split into two groups.
Many stores use the entire population of customers, not a targeted segment, to perform this operation. This is because they want the entire population to receive the sales messaging.
Time of Year and Duration
The emails were sent daily throughout the month of November when shoppers begin shopping for the upcoming holiday season.
Cost of Dresses
This site sells only dresses, and there were 2,000 dresses at the beginning of the campaign. All dresses at checkout were adjusted to $100. No promotional code was necessary.
The wholesaler's original cost to the store was $69.44 for each dress. The store marked up the dress 80 percent to $125. For this promotion, dresses are discounted 20 percent to $100 each.
Medium Type
Shoppers can buy these dresses only on the Internet.
Steps to Assess the Success of the Email Campaign
Both the no model and the model key performance indicators were included. These key indicators involved who was the targeted group, how many emails were sent, when the customer would receive the email, what percentage received and opened their emails, and the conversion rate calculated.
List size: A random sample of 15,000 women per day with a history of buying dresses at the company’s stores is used. The same women are targeted every day.
Delivered rate: Approximately 90 percent, or 13,500 women, received the email per day.
Open rate and kept subscription to emails: Of the 13,500 women who received the email, 15 percent (2,025 women) opened their email per day.
Click rate: Of the 2,025 women who opened the email, 23 percent ( 461 women) clicked through to the dresses per day.
Conversion rate: Conversion = (Number of orders)/Sessions*100 /day = approximately 22/461*100=4.77%/day. The session is over once the browser is closed. Calculate the conversion rate per day. After 30 days, all orders were processed to determine the campaign’s effect.
List size: A random sample of 15,000 women per day with a history of buying dresses at the company’s stores. The same women are targeted every day.
Delivered rate: Again, approximately 90 percent, or 13,500 women, received the email.
Open rate and kept subscription to emails: Of the 13,500 women who received the email, 13 percent (1,755 women) opened the email per day.
Click rate: Of the 1,755 women who opened the email, 27 percent (468 women) clicked through to dresses per day.
Conversion rate: Conversion = (Number of orders)/Sessions * 100/day = approximately 26/468*100=5.56%/day. The session is over once the browser is closed. Calculate the conversion rate per day. After 30 days, all orders were processed to determine the campaign’s effect.
Statistics Conducted: Results and Explanations
An independent t-test was conducted here to determine whether there is a difference between the means of the two groups’ conversion rate, revenue, dresses sold, and ordered dresses. A two-tailed test was used for all the t-tests. A Pearson correlation test was conducted for conversion rate and revenue to see if there was a relationship.
Independent T-Test 1: Conversion Rate Between Models and No Models
- Step 1: The Hypothesis, or the Reason for the Business Question:
Null hypothesis: μ1 = μ2
Meaning: There is not enough evidence to determine if there is a difference in the means of the conversion rate between the model and no model conditions.
Alternative hypothesis: μ1 ≠ μ2
Meaning: There is a significant difference in the mean conversion rate between the model and no model versions.
Step 2: Confidence Level: Alpha = 0.05
Step 3: See “Mathematical Operations and Statistical Formula” in Appendix B.
Step 4: Results: There was a significant difference between the model/no model’s mean rate conversion rate.
Step 5: Descriptive Analysis: There was a significant difference between the mean conversion rate of the model (5.56) and the no model (4.77). There was a higher conversion rate for the models. In other words, more shoppers bought when there was a model. The shoppers saw a picture and liked how it looked on a real human.
Independent T-Test 2: Revenue Between Models and No Models
- Step 1: The Hypothesis, or the Reason for the Business Question:
Null hypothesis: μ1 = μ2
Meaning: There is not enough evidence to determine if there is a difference in the means of the revenue between the model/no model conditions.
Alternative hypothesis: μ1 ≠ μ2
Meaning: There is a significant difference for the mean revenue on-site between the ones that used models or no models.
Step 2: Confidence Level: Alpha = 0.05
Step 3: See “Mathematical Operations and Statistical Formula” in Appendix B.
Step 4: Results: There was a significant difference between the revenue of the model and no model groups.
Step 5: Descriptive Analysis: The model group had a higher mean (3,583) than the no model (2,603) group, and there was a significant difference between the two means. This shows that when you use a model, there are better financial gains than when a model is not used.
Independent T-Test 3: Dresses Sold Between Models and No Models
- Step 1: The Hypothesis, or the Reason for the Business Question
Null hypothesis: μ1 = μ2
Meaning: There is not enough evidence to determine if there is a difference in the means of dresses sold on-site between the campaign that used models and the one that used no model.
Alternative hypothesis: μ1 ≠ μ2
Meaning: There is a significant difference for the mean dresses sold on-site between the campaign that used models and the one that used no model.
Step 2: Confidence Level: Alpha = 0.05.
Step 3: See “Mathematical Operations and Statistical Formula” in Appendix B.
Step 4: Results: There was a significant difference between the models and no models on mean dresses sold.
Step 5: Descriptive Analysis: The groups had a significant difference in their mean dresses sold between the model (36) and no model (26). Therefore, the presence of a model or no model did positively affect the average number of dresses sold.
Independent T-Test 4: Orders of Dresses Between Models and No Models
An independent t-test was conducted to determine if there is a difference between the mean number of dresses ordered between the two groups.
- Step 1: The Hypothesis, or the Reason for the Business Question:
Null hypothesis: μ1 = μ2
Meaning: There is not enough evidence to determine if there is a difference in the mean number of dresses ordered between the campaign that used models and the one that used no model.
Alternative hypothesis: μ1 ≠ μ2
Meaning: There is a significant difference in the mean number of dresses ordered between the campaign that used models and the one that used no model.
Step 2: Confidence Level: Alpha = 0.05.
Step 3: See “Mathematical Operations and Statistical Formula” in Appendix B.
Step 4: Results: There was a significant difference between the number of dresses ordered between the models and no models versions.
Step 5: Descriptive Analysis: The groups had a significant difference in their mean number of dresses ordered between the model (26) and no model (22). Therefore, the presence of a model encouraged the buyer to buy more dresses per order.
Pearson Correlation by Model: Relationship Between Conversion Rate and Revenue
- Step 1: The Hypothesis, or the Reason for the Business Question:
Null hypothesis: p = 0
Meaning: There is not enough evidence to assume that there is a relationship between conversion rate and revenue.
Alternative hypothesis: p ≠ 0
Meaning: There is a significant correlation between conversion rate and revenue.
Step 2: Confidence Level: Alpha = 0.05.
Step 3: See “Mathematical Operations and Statistical Formula” in Appendix B.
Step 4: Results: There was a significant positive relationship between conversion rate and revenue when a Pearson correlation was conducted separately for both the models (0.75) and no models (0.62).
Step 5: Descriptive Analysis: This shows that whether a model was present or not, as the conversion rate increased, so did the revenue. This is what the analyst would hope to find.
Conversion by revenue for model
Conversion by revenue for no model
Sell-Through Rate for Model and No Model
The formula for sell-through rate gauges how fast the company sells its products. It aids the company to predict its demand in that product. Sell-through rate measures the percentage of inventory that is sold compared to the inventory that is sent from the manufacturer. Most companies look very carefully at this percentage. This percentage lets the company know how many products to purchase or that there is a problem selling certain products (although it does not indicate what the issue is).
Sell-through rate = units sold/(units sold and units on hand) × 100
Model = 1,077 dresses sold/2,000 dresses total × 100 = 54% for November
No model = 781 dresses sold/2,000 dresses total × 100 = 39% for November
Dresses not sold = 2,000 – 1,077 – 781 = 142. Take 142 dresses not sold/ 2000 dresses × 100 = 7% for November.
Average Order Value for Model and No Model
The average order value is a metric that tracks every dollar spent by customers with every purchase the customer makes. This is a benchmark, and it indicates that as the average inventory value increases so does the profits for the company.
AOV= Average purchase = total revenue/ total orders
Model = $107,500/780 = $137.82 or $138 for November
No Model = $78,100/661 = $118.15 or $118 for November
The average order value for the customer is higher for the model than the no model, which is to be expected because with the model the customer has an opportunity to see the garment worn on a model and has a better idea how the garment would look on them.
Note: To have a large enough sample, it is typically recommended that 30 or more data points be used for each type of model group. Depending on the size of your database, more sophisticated sample size calculations can be considered.
Total Metrics on Key Performance Indicators for Email Campaign
The key performance indicators (KPIs) are important vital factors that allow a company to improve their strategies and operations.
Basic Metrics for Email Campaign
Mean Type of Model | Conversion Rate per Day | Average Click Rate | |
|---|---|---|---|
1 (no model) | 4.77% | 23% | |
2 (model) | 5.56% | 27% |
The conversion rate is the number of conversions divided by the number of visits multiplied by 100. The no model conversion rate would be approximately 0.0477/day, or approximately 4.77 percent/day, and for the model the conversion rate would be approximately 0.0556/day, or approximately 5.56 percent/day. November has 30 days. Therefore, 30 conversion rates and 30 response times for each type of model are needed.
Average Click-Through Rate
The average click rate is the average number of times customers will click on the email link to the total number of customers who received the email for the email campaign.
Type of Model
1 = no model (range of 40; it is between 100 and 140 avg click rate and has a bigger standard deviation).
2 = model (range of 10; it is between 115 and 125 avg click rate and has a smaller standard deviation).
Profit per Dress
Revenue or Profit from Campaign
Dresses sold times the cost of the dress | |
|---|---|
Model | No Model |
30.56 dress cost × 1,077 dresses sold | 30.56 dress cost × 781 dresses sold |
= $32,913.12 | = $23,867.36 |
Profit per dress = sales price of dress-original wholesale = $100 – $ 69.44 = $30.56
Total profit from campaign = $56,780.48 = $32,913.12 + $23,867.36
ROI and ROAS
Return on investment (ROI) and return on ad spending (ROAS) are two essential metrics for measuring the results and profits for the campaign that is run. ROI is the net profit divided by the net spending multiplied by 100. ROI includes the cost of software, designs, and people.
ROAS is revenue made from ads divided by advertising spending. ROAS does not tell you if the paid advertising is effective, but it explains if the ads are effective in driving clicks and revenue.
The higher the ratios for both the ROI and the ROAS, the better. This means the company is gaining a higher percentage in revenue than cost for the ROAS and also bringing in more profit than investment for the ROI.
Here are the ROAS and ROI formulas for the model and the no model:
ROAS = revenue derived from ad campaign/cost of ad campaign
ROI = profit derived from ad campaign/total cost of investment from campaign
Model No Model
ROAS = 32,913/6,570.4 = 23,867.36/5,966.84
Ratio 5:1 4:1
(The example reads as follows for the model: for every $1 spent on the ad campaign, $5 is gained in revenue.)
ROI = 32,913/8,213 = 23,867.36/7,955.79
Ratio 4:1 3:1
The ratios for the model are higher than the ratios for the no model version.
Summary and Discussion on Results
We had a business question of, does showing the model encourage the buyer to purchase more? Using the A/B testing method, we showed that it does. This was demonstrated because the model group had statistically significant different means that were larger for conversion rate, orders made, dresses sold, and revenue acquired than the no model group.
The other business question was, does the conversion result in more revenue? This showed a significantly positive association for both the model and no model groups, which is also a good sign. This means that as conversion rate increased, so did revenue in the positive direction. If there was a negative correlation this would mean that discounts were driving the increase of conversion rates. There were not many dresses left after the campaign, and the sell-through rate for the model was better than the no model version. The model performed better than the no model version for the sell-through rate, AOV, average click rate, and profit. The ROI and ROAS were better for the model version than the no model version. These findings are to be expected because the buyer was given clearer information about what they were purchasing.
Thoughts for Further Analyses
This email campaign was successful and, for the most part, came out as expected. As anticipated, the model group mean revenue, conversion rate, orders, and dresses sold were all more than the no model version. And for both the model and no model versions, there was a positive relationship between revenue and conversion rate. This means the more interesting the merchandising and more information the buyer has, the more they could picture the dresses on themselves and decide to buy them.
Moreover, the brand values that the store is selling are key to understanding and analyzing the results of the email campaign. It is imperative to keep the brand desirable and relevant or, at least, consistent and sustainable in order to retain and continue to increase the number of loyal customers. The object is to sell products and increase revenue while controlling or minimizing costs and returns, while also reinforcing brand value such as best price, best quality, or some other core brand feature.
Summary
The focus of the email program needs to be on retaining customers in the database and on growing the database with potential and new customers. Emails tests such as the one shown in this chapter help to optimize merchandising to maximize conversion rate and sales from each campaign. To increase direct sales at the program level, the store manager could use email sign-up on the website to retain new visitors and optimize copy with search keywords to help attract potential customers.
The astute manager will need to run year-over-year predictive analysis, using variance and percent variance to make sure the company is maintaining and increasing its revenue based on the metrics of the key performance indicators, which will be reviewed in the next chapter. Month-over-month analysis may be conducted as well, but keep in mind that sales are different at various times of the year, and each month has a different number of days. The next chapter will cover how to create scenarios for anticipating sales and forecasting into the future.