Perform multivariate testing
Multivariate testing is a method used to evaluate two or more variables in the checkout process to see which one leads to higher revenue or conversion rates. Even small adjustments, like modifying a button or changing a color, can impact conversion rates. Testing these changes through multivariate testing is essential for optimizing your checkout process.
If you're unsure of the most effective way to present your checkout, set up a multivariate test. For more details on the benefits of this approach, refer to Best practice: Test with MVT campaigns.
How multivariate testing works
You set up multivariate tests in the MVT Campaign section of our web admin tool or the MVT Campaign portal in Commerce Assistant (CA) by creating different versions of the checkout process, called candidates. When a customer starts the checkout, one of the candidates is randomly assigned. This random assignment ensures that factors like the day of the week or time of day don't affect the test results. However, it’s recommended to avoid running tests on bank holidays.
Each customer is assigned a cookie so they will consistently see the same version of the cart if they return.
Once you have a sufficient number of completed paid orders, you can compare the total revenue generated by each candidate using reports. The number of orders needed to determine a clear result depends on the confidence level, which reflects how certain you can be about the results. A 95% confidence level means you can be 95% sure of the result, while a 99% confidence level means 99% certainty. Most researchers use a 95% confidence level. This level is calculated using a formula based on uplift (standard deviation) and normal distribution. When you reach a confidence level of 95-99%, the uplift is statistically significant and can be trusted.
Afterward, you can select the winning candidate to implement in the checkout process. The MVT campaign can also continue routing traffic to the selected candidate if needed.
Variables
A variable is the element you want to test in your checkout process. This could include:
- Different checkout configurations for the same market
- Price points
- Opt-in vs. opt-out options
- Page layout elements
- Icons, buttons (such as button size and color), or other visual elements
With multivariate testing, you can test multiple variables simultaneously, unlike traditional A/B testing, which limits you to two.
For the most accurate results, each candidate should test only one variable. For instance, if you're comparing opt-in vs. opt-out, one candidate should feature the opt-in option, while another features the opt-out.
Weight
Each candidate is assigned a weight, which determines how often it is randomly shown to customers. For example, you can direct 40% of traffic to Candidate A and 60% to Candidate B. The weight distribution should align with your business objectives and testing goals. However, it’s recommended to keep the default weight value of 100 for each candidate, ensuring an equal distribution across active candidates.
For example, if you want an even 50/50 split, both Candidate A and Candidate B would each have a weight of 100. For a 66.6/33.3 split, Candidate A would have a weight of 100, while Candidate B would have a weight of 50.
Best practice: Test with MVT campaigns
When you are optimizing the checkout process and performing multivariate testing, you can choose from several best practices in order to achieve the best test results:
Test similar variables
For clear test results, test similar variables, such as:
- A button with two different colors
- An opt-in vs. an opt-out
- A header vs. no header This ensures you have measurable results. If you test different variables, such as a button color change vs. an opt-out, it will be impossible to tell which item triggered the purchase.
Test one change at a time
Test one change to your checkout at a time to make it clear what is driving conversions. For example, you want to test a button color change and an opt-in vs. opt-out offer. Instead of testing both changes at once, first test the button color change. After you are satisfied with those results, you can then test the opt-in vs. opt-out.
Test candidates at the same time
A common testing approach is to run one candidate for a set period (at least one full week, though 14 days is recommended), then switch to a second candidate for the next period. However, testing all candidates simultaneously is more effective, as it avoids external influences like seasonal trends from affecting the results. The MVT campaign randomly assigns candidates, helping to minimize the impact of such external factors and provide more accurate results.
If you have one MVT and two candidates with equally 100 sessions, Candidate 1 wins. In order to test Candidate 1 against another candidate, deactivate Candidate 2 and set up a new candidate for the same MVT for testing. Setting up a new candidate prevents wrong testing results.