Internet sites are flexible things. A change of code here, a new graphic there can have a site looking very different in a short space of time. So why, if it is so easy, do we not see our sites changing from day to day? The main reasons are continuity and branding, changing a site constantly is going to leave your customers confused and not necessarily very friendly towards your site or your company. However in order to make improvements to a site change becomes necessary.
The hassles of designing alternatives and struggling to manage and analyse the impact of any changes put many off the concept of testing things out on a website in anything other than a conservative manner. Ideally, one would make changes and expose a small percentage of their customer population and see how they react to the new changes, this is what we would call a “test”. Many web marketers have not felt empowered to conduct changes and tests upon their sites.
This is unfortunate considering the web is such a good mechanism for testing the interactions between customer and business and that many small changes (links, pictures, banners etc) can be introduced with very little negative feedback from customers.
This concept was appreciated by some in the industry many years ago but it’s the recent rise of the web analytics vendors who are looking to enhance their products by incorporating automatic testing that have jumped on the idea of A/B split tests or multivariate testing that brings this technology to the public eye. Multivariate testing is actually a very broad term covering a very wide range of techniques. Multivariate testing as used by web analytics providers seek to provide several versions of a page (tests) to your customers and “decide” what is the best layout by identifying which version has a significant difference to other versions and provides a greater conversion. Rather than A/B testing its more like A/B/C/D/E testing.
The idea behind vendors’ current approach to testing is that you set up an “experiment” on your web site. In this experiment you highlight particular areas that you might like to change (this could be the position of a button, the colour of an advert, the words in some text, the term paid for at your favourite search engine, anything really) and then identify some metric which you wish to improve (click through rate, conversion, purchase rate etc). You simply make the changes and record whether the customers shown the changed website score any higher on the chosen metric than those who weren’t.
This is a wonderful introduction of a very old strategy of testing and analysing observations. It does however have some limitations unless applied in an appropriate manner.
Imagine the situation, you’ve got a front page and only space for one banner. How do you choose which of your two favourite banners to place in your top advertising spot?
Let’s help you in your decision by introducing some facts and figures through using a three stage example.
1) Both banners have previously occupied the premium front spot and their conversion rates were dramatically different.
Banner | Average Conversion #’s per month |
A | 1000 |
B | 15 |
Whoa! There’s no choice here! There’s clearly a significant difference in conversion rate of the banners and I know which one I’d be picking, A of course!
2) Ah, but wait, there’s a little piece of information I forgot to mention. Banner A and Banner B do not advertise the same product. Banner A advertises Product A that retails at 12 dollars (10 dollars profit) and Banner B advertises Product B that retails at 800 dollars (600 dollars profit). Now how is our decision looking?
Banner | Average Conversion #’s per month | Profit per sale | Overall Profit |
A | 1000 | 10 | 10000 |
B | 15 | 600 | 9000 |
Now things are getting interesting. The difference in profit to the website (and therefore the company) between the two banners is lot closer, possibly not even significant if it were statistically tested, however Banner A is still the one I’d be going for.
3) So now the last surprise. It has been shown through previous analysis that those who buy Product B are 80% likely to go on and purchase further products during the month to the average price of 500 dollars and an average profit of 300 dollars. However those customers that purchase Product A have never been shown to purchase any other product.
So let’s look at our table now.
Banner | Average Conversion #’s per month | Profit per sale of Original Product | Profit of Additional Purchases | Profit |
A | 1000 | 10 | 0 | 10000 |
B | 15 | 600 | 0.8 * 300 | 9000 + 3600 = 12600 |
So there you have it, it seems like a better idea to place Banner B on our front page, despite the fact it converts a lot less people.
So let us review what happened there over the 3 stages.
1) When we were basing our decision on solely number of conversions and our Banner A won.
2) When we began to look at our banner choice based on value associated to the product that was being advertised and again our Banner A won but the choice was a lot harder.
3) When we began to look at our banner choice based on value of each customer and saw that actually, Banner B would be our preferred choice.
The conclusion that one should use customer value as a guide to deciding which advert should be displayed is not only applicable when comparing adverts of differing products. The same also goes for looking at adverts for the same product. What one has to gauge when doing this kind of testing is whether the different adverts are appealing to different segments of the market place and if profits is the main goal (rather than say, volume) then one should look at the value of these segments in conjunction with any rates of conversion.
How does this compare to the kind of A/B or Multivariate testing offered by web analytics vendors?
Well, first of all there is much concentration on the metric of conversion. The act of someone converting is currently the holy grail of web analysis. This is simple to do and quick to perform. It’s a very quick step from recording the number of conversions on a banner through a simple calculation in order to get a measure of significance. But as shown above, just indicating there is a difference between outcomes in an experiment does not identify a winning strategy for your website, it explains what is happening on your website in response to changes but not why, and it’s the “Why?” questions where all the real value is at.
When this kind of multivariate testing is done in laboratories across the land, scientists attempt to establish a control against which you can measure changes. The majority of current web analytics vendors don’t appreciate or allow you to identify a control case/situation. They work on the assumption that everyone coming on your website, be it through a paid click, organic search or just simply typing in the appropriate URL are all trying to do the same thing and that thing is to buy (or convert).
In assuming that everyone is the same, a “significant” difference in uptake across banners is interpreted as a “better” converter rather than a reflection of different feelings or intentions of the customers that use the site and this means that a greater understanding of why some banners “work” and others don’t is simply lost.
In very broad terms customers come to your site for a few simple reasons.
Buying - or performing some activity that equates to purchasing, such as requesting to view specialised content
Research prior to Buying (see point one, where a customer does not “commit” to your site and is merely trying to identify what you offer and possibly compare it to other sites)
Browsing your site – this is the equivalent to window shopping, wondering around with no intention to commit or purchase.
The best a website can do is try to work out what a customer wants from them and act accordingly. Hopefully you can convert the ones that are on the look out to buy as these are our “convertibles”. Then customer behaving like 2) and 3) need to be treated differently to each other (and those customers in category 1). Efforts must be taken to ensure that 2) come back again, not that they necessarily purchase today. The majority of your customers will be in these later 2 categories.
Although having the highest converting banner has obvious benefits, it only works on a small percentage of your population. What you should be developing is clear strategy that that moves customers from one activity level to another. This again is acknowledged by web analytics vendors who enjoy demonstrating their ability to draw activity “funnels” but again these are only reflections of what is happening on your website and offer little or no insight as to why. Good segmentation changes all of that, good segmentation will allow you to understand your customer base better and make changes in your site that allows you to interact with them differently.
Using Segmentation as Selection Criteria for Better Testing
In order to nullify any effects that may appear from customers belonging to different segments or having different intentions you could do a number of things.
Run the multivariate test as usual, analyse the customers who took the test and try to get to the root of “why” one design was better than any other in post analysis and hope you can find some satisfactory results. Typically one would use some kind of profiling technique for the different cases, such as logistic regression (also classed as multivariate analysis, but let us not confuse things) or a decision tree algorithm.
Pre-select your groups of people before the test so they all conform to the same customer segment e.g. High Value, Frequent Visitor, Regular Buy etc. And then perform the test as usual. Any differences discovered can then be attributed to the power of the banner for that particular segment.
Run your tests over a set of customers that you have already segmented. Each segment should be exposed to each outcome in the multivariate test and compared against each other.(assuming that you can gather enough cases for each segment to make fair comparisons)
What you are trying to achieve by segmenting your customers is a level platform from which you can assess the impact of any changes you make on your site.
Tips for Segmentation
Assuming you have chosen (wisely) the third option above, when performing online segmentation it is generally advised to segment based on customer behaviour or their value. (Demographics could be included too but most businesses don’t have this information for many of their site visitors unless they have a good registration process that most customers sign up to)
Segmenting behaviour based on pages seen previously/products purchased etc. is an extremely good way of identifying the “intention” of the customer prior to being exposed to the various designs. Behaviour is the best technique to use if you don’t have much information about the customer, perhaps they are new to your site, perhaps they have not bought anything yet or indeed bought only a few specific items.
Segmenting value based on broad metrics (such as RFM – recency frequency, monetary value) is a popular method in most traditional bricks and mortar businesses. The overall RFM metric is often split into various subsequent groups, quartiles, deciles etc. It is good because it tells you a lot about “who” a customer is terms of value. Using value as a segmentation metric is a good way of identifying a groups’ likelihood to do anything (buy a product, click banners, follow links). This is useful when analysing customers with whom you have a relationship with over a good period of time.
If you’re new to segmentation, pick the measures that you feel most comfortable with and that reflect the kinds of customer that you have. If you’re an experienced segment-er you may want to look into combinations of behaviour, value and even demographics however it will begin to get complicated to do this by hand as you add more attributes. I’d recommend using some form of algorithm that will help you identify your segments accurately (either from the world of statistics or data mining) and have it report back in the segments it finds in a manner that is easy to interpret and actionable.
There will be segments of which you know very little, those that have just appeared or the first time on your website. These people should not be ignored because you think you don’t have much behavioural information. It is most likely the lack of behavioural history that will allow us to put the in a particular segment as opposed to your regular customers. Key attributes for these customers will be where they come from (search engine) how they got here (search term) and what they did in their short visits.
Once you have your segments defined, you simply apply your multivariate test and observe the results to see if there are any differences. The differences you will see will fall into three types of outcomes and each outcome provokes a different action.
If there is a difference between the uptake of the designs and no difference across the segments then you have a winner. You have clearly identified a better performing attribute (banner/link etc.) on your website and you should go ahead and make the change.
If there is a difference across segments then you have different types of customers of different value using your website and you should look into the possibility of personalising your website, tailoring towards each groups’ needs.
If you have no difference between designs or across segments then the change is not important.
This article has highlighted the need for A/B or multivariate testing to be done in conjunction with a selection procedure (in this case segmentation) in order to produce results a business can be confident in.
Without the ability to select your customers in some way and compare their different reactions you will always be wondering why any design is considered better than any other by the testing applications. This kind of statistical testing introduced by the web analytics vendors is a great step to indicating “what” might be an improvement to any website, let us move forward and start asking “why” it is an improvement.
