A complete guide to A/B testing

Introduction to A/B testing

A/B testing is basically the comparison of two different versions of the same web page to conclude to which one is better working. This allows you to formulate better marking strategies as it has valuable answers to your business questions generating better revenue from your current clients.


It divides the web page visitors into two halves displaying each half one version of the web page and vice versa. By doing so, the page showing better rates of conversion can be identified. The primary version is referred to as ‘Control’ while the secondary version is referred to as ‘variant’.

The version declared to be of better interest to the customer is labelled as the control version and another variant version is designed.

To gauge success conversion rates are a very immature parameter. The flow of cash with regard to conversions is very essential.

A/B/N testing

This is an enhanced method of comparing multiple versions of the web page but uniformly dividing the audience. One version is called the control while the other ones are labelled as variants.

Duration of A/B testing

It is recommended that the A/B test should be run for two business cycles. The sample size determined before the test is begun is also to be kept in mind. The duration is to understand all the accounts of the buyer’s interest along with an account to various traffic sources like Google or your Friday newsletter.

Make sure you run the test the entire week as different days and different times have different traffic. You wouldn’t want your calculations to go wrong for apparently not utilizing the test to its right capacity.

Purpose of using A/B testing

The main purpose of using the A/B test is to attract the traffic in the best manner to generate better conversions. After all the investment and effort you make to get a visitor to your webpage shouldn’t go wasted. This test helps you in realizing how to cash your visitors best. Your gain is not dependent on the winning or losing of your test as either way it leads to valuable insights.

Besides working on other factors like your products and customer care are too of greater importance.

Don’t run A/B tests

A/B testing is only effective if you’re website has high traffic. Your ROI to A/B testing is only valued if a good amount of customers use your webpage. If you don’t expect a lot of people visiting your page you need to look for alternates like taking feedbacks from your customers directly.

Running the right tests

To design tests for better results here are few key points to keep in consideration

  1. Technical analysis
  2. Survey’s on and off-site
  3. Questioning your customers
  4. User testing
  5. Analytics analysis

Prioritizing frameworks

The three main prioritization frameworks are ICE (impact, confidence and ease), PIE (potential, importance and ease) and PXL (potential, importance and ease). These tests involve your judgment thus when a large number of people conduct this test the rankings may vary. It allows setting better protocols and limitations for better results.


You can categorize your approach to improve the idea of your test. A simple categorization of implement, investigate and test will make your job a lot easier. Implementation is basically doing it to see what potential it has. Investigation is the thought process required and testing is the conclusion of the idea to be solid.

Role of statistics in A/B testing

Alex  Birkett, Growth Marketing Manager at Hubspot, explains statistics to be more than just a number when we relate it to A/B testing. It is very effective in reducing the risk factor. He believes that the mean, variance, regression and other values of the representative sample are to be studied to set better parameters to minimize human errors.

The founder of the CXL institute Peep Laja subjecting to statistical significance elaborated about the two conditions to be understood. One of them he said is the determination of the right sample size while the second condition is to ensure the avoidance of statistical population.

Analyzing A/B test results

According to Krista Seiden, analytics advocate and product manager at Google, “the best way of improving A/B test results is by learning from the losers. Losers here are the failed tests.” She has had a routine of publishing failure reports just to analyze what mistakes were made and what can be done to rectify them.

A/B testing is a very statistically analyzed survey which has a lot of math and complexity to it. It is not something you expect to be an expert of overnight. It requires a vision to understand the direction of the results to learning ways of improving it. There’s loads of work needing o be done before deciding to opt for A/B texts. From collecting data to determining the number of variants appropriate to use and a lot more other technicalities has all to be brought into consideration. If A/B testing is what you’re looking for you need to start studying about its method and purpose right away along with its integration to practical statistics. Is it effective? If organizations like Google and Hubspot are working on it, it sure is.


3Bet Media is a Digital & Growth Hacking Marketing Agency, London based.
Contact us for a tailored Growth Hacking solution at hello@3betmedia.co.uk.

Source: Shopify, Google.

Posted on February 13, 2018 in Digital Marketing, Growth Hacking

Share the Story

Back to Top