Manage an MVT Campaign

You can use the MVTClosed Multivariate testing (MVT) is a technique for testing two or more different variables in the checkout process to determine which variable creates more revenue or a higher conversion rate. Campaign portal to add, view, and edit MVT campaigns used for multivariate testing. You can also use this portal to route customers by region, language, browser, or operating system without performing testing.

Different versions of the checkout process, called candidates, are set up. When a customerClosed An individual or business purchasing your product or service by placing an order through Cleverbridge. The customer is the end user of this product, as they are not allowed to resell the purchased products or services. ​ A customer is unique per client. If a customer purchases products or services from two different clients, there are 2 separate records of said customer.​ begins the checkout process, one of the candidates is randomly assigned if the candidates have the same link audience.

A candidate ranking indicator with stars displays for active, eligible candidates and shows you when a candidate becomes statistically significant and should be analyzed.

For information on how to implement an MVT campaign, see Basic Steps for Implementing an MVT Campaign.

To successfully manage an MVT Campaign, complete the following tasks:

Step 1: Add an MVT Campaign

In the main menu, select Tools > Add MVT Campaign. The MVT Campaign portal opens

  1. Enter a unique name for the campaign.
  2. In the Display URL field, enter a unique identifier to be added to the end of the display URL. The first part of the URL is pre-filled. This URL is used as the campaign URL and is visible to customers.
  3. Click the Copy Copy button to copy the URL and paste it in your website. The Active checkbox defines whether or not this campaign is actively used for testing.
  4. Add candidates to the campaign. For more information, see Step 2: Add an MVT Candidate.

Step 2: Add an MVT Candidate

One candidate is automatically added for you. You can add as many candidates as necessary for the campaign. These candidates are randomly selected for customers during testing when the display URL is clicked. To add an MVT candidate, complete the following:

In the Candidate Name field, enter an identifier for the candidate that uniquely identifies what this candidate represents. This identifier is also used for reporting in the BI. Do not change the candidate name after you created the candidate. Each candidate gets a unique ID. It might happen that a name change results in two candidates with one ID.

The Active checkbox to the left of the Candidate Name field defines whether or not this candidate is actively used for testing.

Create the URL in one of two ways:

Create the URL in the Link Generator

  1. Click the View in Link GeneratorNew link button to the right of the Destination URL field. The Link Generator contextual panel opens.
  2. Create the link in Link Generator.
  3. Copy the link.
  4. Return to the MVT campaign.
  5. In the Destination URL field, right-click and select Paste. The URL is pasted.

Manually Create the URL

You can manually create the URL. For information about the Cleverbridge URL structure, see Control & Configure Your Checkout Pages Using URL Parameters.

You can determine how often your candidates are used in the campaign using the Candidate Weight field. If you give one of your candidates a higher number in the Candidate Weight field, this means that the candidate will be assigned to more customers.

Example

In MVT Campaign 1, there are two candidates and they both have a weight of 100. Both candidates are evenly distributed for customers in a 1:1 ratio.

In MVT Campaign 2, there are three candidates and they all have a weight of 100. All candidates are evenly distributed for customers in a 1:1:1 ratio.

In MVT Campaign 3, there are two candidates. Candidate A is distributed twice as much as Candidate B. Candidate A has a weight of 100 and Candidate B has a weight of 50. They are distributed to customers in a 2:1 ratio.

In MVT Campaign 4, there are three candidates. Candidate A has a weight of 100, Candidate B has a weight of 80, and Candidate C has a weight of 20. They are distributed to customers in a 5:4:1 ratio.

By default, the Candidate Weight field sets up the ratio in which your candidates will be distributed. If you want to equally distribute all of your candidates, keep the weight at 100 for each candidate. If you want to distribute a candidate less than the other candidates, make its weight less than 100.

On the other hand, if you want to use the Candidate Weight field to specify the percentagein which particular candidates are assigned, ensure that the combined sum of the weights of your candidates always equals 100.

Example

A fictional company called Shieldware creates an MVT campaign with three candidates. Candidate A is to be assigned 50% of the time, Candidate B is to be assigned 35% of the time, and Candidate C is to be assigned 15% of the time. To achieve this, they give the candidates the following weights:

Candidate Name Candidate Weight
Candidate A 50
Candidate B 35
Candidate C 15

If Shieldware ever adds/deactivates a candidate, then they must adjust the weights of the remaining Active candidates and ensure that their combined sum equals 100 weight .

To Select a Weight:

There are three ways to select the weight:

  • Click the circle and drag it to the appropriate weight.
  • Click the up and down arrows to the right of the field.
  • Click inside the field and enter a number.

Define your audience for the candidate . Select from the following options:

Important

By default, customers are always routed to the very first candidate assigned to them, even if they return to the checkout process using the same computer in a different situation (for example, they access the checkout process from different countries). Therefore, if you want to give customers a fresh candidate everytime they access the checkout process, you must use the Reset MVT Cookie Parameter (&resetMvtCandidate). For more information, see Cookie Settings.

Note

When a URL is submitted that lacks a particular option (for example, Browser), and you have created four MVT candidates (three candidates with a particular option and one candidate without it), then only the candidate without the option will be considered.

Option Description
Region

Define a specific region for the candidate. This can be used to test different designs or configurations in different regions. To add a new region, click the add icon button to the right.

Example

Candidate A is displayed in Germany. Candidate B is displayed in Switzerland.

Tip

To ensure the rest of the world has access to the standard checkout process, add another candidate and specify a region that covers all other regions not included in your region-specific candidates.

Note

The Region option has a higher priority than the Language option. Therefore, if you defined two candidates as follows:

  • Candidate A: region = US
  • Candidate B: language = Spanish

then a customer who accesses the checkout process with Spanish language settings and an IP address from the US will be given Candidate A.
Language Define a specific language for the candidate. The language is selected based on the customer's default browser language.
Browser Define a specific browser for the candidate.
Operating system Define a specific operating system for the candidate.
Parameter Name

Define an additional parameter for the candidate (for example, audience=google)

Example

The fictional company, Shieldware, wants to conduct a single MVT campaign, but they want to test different elements on different types of customers:

  • For all customers who initially access their checkout pages through the Shieldware website, they want to conduct an A/B test evaluating the success of the 2016 checkout-page layout compared to the 2017 layout.
  • For all customers who initially access their checkout pages through marketing emails, they want to conduct an A/B test evaluating the success of pre-filled customer details compared to blank customer details.
  • For all customers who initially access their checkout pages through a search engine, they do not want to conduct an A/B test.

To accomplish this, Shieldware creates an MVT campaign named “Traffic” that contains five candidates. They name and define the candidates as follows:

Candidate Name Definition
Audience 1A

Customers who initially access the checkout pages through the Shieldware website and receive the 2016 checkout-page layout.
Audience 1B

Customers who initially access the checkout pages through the Shieldware website and receive the 2017 checkout-page layout.
Audience 2A

Customers who initially access the checkout pages through a link in a marketing email and receive pre-filled customer details.
Audience 2B

Customers who initially access the checkout pages through a link in a marketing email and receive blank customer details.
Audience 3

Customers who initially access the checkout pages through a link in a search engine and receive no test checkout pages.

They insert the following values in the Parameter name field for each candidate:

Candidate Name Parameter Name
Audience 1A shieldware
Audience 1B shieldware
Audience 2A email
Audience 2B email
Audience 3 search

When customers access Shieldware’s checkout pages for the first time, these parameters wil route them to the appropriate candidates. For example, if a customer initially accesses the checkout pages through the Shieldware website, they will be randomly assigned to Audience 1A or Audience 1B.

https://www.shieldware.com/864/purl-anti?audience=shieldware

On the other hand, if a customer initially accesses the checkout pages through a marketing email, they will be randomly assigned to Audience 2A or Audience 2B.

https://www.shieldware.com/864/purl-anti?audience=email

Tip

If you want to use the Parameter Name field to route customers to particular candidates, you must use the same values as for your traffic sources. Contact Client Experience for additional information.

Tip

To ensure that MVT filtering is functioning properly, you should also add an additional or fallback candidate with no value in the Parameter Name field.
Affiliates

Select whether or not to restrict the candidate for affiliates.
HTTP referer pattern

Define an HTTP referer pattern for the candidate.

Tip

The pattern must be written as a regular expression.

Example

The fictional company, Shieldware, has links to the checkout process on its landing page, its home page, and one of its content pages. They want to test different elements on different customers, depending on through which web page they accessed the checkout process.

To accomplish this, Shieldware creates an MVT campaign named “Referer” that contains three candidates. They name the candidates as follows and define one HTTP referrer pattern for each one:

Candidate Name HTTP referrer pattern
Audience 1 (https://(.*?)/landing)
Audience 2 (https://(.*?)/home)
Audience 3 (https://(.*?)/content)

When customers are forwarded to Shieldware’s checkout pages through one of these web pages, they will be routed to the appropriate candidate.

Enter detailed notes about the candidate's purpose.

Tip

As the test progresses, use the notes field to keep track of the testing results, such as why a candidate was successful or not successful. That way you can keep a history of changes.

  1. To add another candidate, click Add Candidate. Another row displays with the new candidate. Repeat steps 1-5.

Step 3: Monitor the Campaign

There are two ways to monitor your campaigns:

The stars on the right-hand side of the candidate name are used to rank the success of a candidate during testing. They provide an indication of when a campaign is statistically significant enough for you to begin analyzing the test results.

Note

The ranking is not intended to provide sufficient information about which candidate performs best. It is only a relative result that needs more evaluation to generate insight for your business.

Rules for Candidate Ranking

The following describes how candidates are ranked:

  • Ranking is eligible for active candidates testing against one another, which means two or more candidates with the same Link Audience settings.

    Example

    If you have three candidates and only two have the same Link Audience settings, then only those two candidates are ranked.

    If you have four candidates and two have the same Link Audience settings, and the other two have a different set of Link Audience settings, both groups of two will be ranked, but in two separate tests.

    Specific example

    You have specified a region for candidate A (for example Europe), then you also need to specify a region for candidate B (for example USA). If you also want to test an operating system, you need to set up one more candidate with Phone as operating system and World as region.

  • The stars initially display as gray and inactive until ranked.
  • Ranking is based on revenue per session. The more stars that a candidate has, the more likely it is that this campaign worked better. However, the confidence level is significant. Contact Client Experience to help you evaluate the results in an Excel sheet. Client Experience can assist you with creating forms to calculate the confidence level.

Note

Always use new candidates when you run tests. If you use an existing candidate to compare to a newly specified candidate, the users per session of the already existing candidate is taken into account. Additionally, the Created field will not change if you deactivate and reactive an existing candidate.

The metrics under the Candidate name can help you determine which candidates to activate permanently . The following explains how to interpret them:

Metric Description
Orders/Sessions

Orders are the number of transactions completed and paid for using the checkout page(s) pertaining to this candidate.

Sessions are the number of visits to the checkout page(s) pertaining to this candidate.
CR

Conversion rate (CR) is the number of orders divided by the number of sessions.

(CR = Orders / Sessions)
Uplift

Uplift is the new conversion rate divided by the control conversion rate minus 1.

(Uplift = (New CR / Control CR) - 1)
Significance Significance is the probability that your A/B test yielded trustworthy results. If you have a statistical significance of 90-100%, you can have a high level of confidence in the validity of your results.

Important

In orderClosed An agreement between a seller and a buyer to exchange goods and/or services for money. An order can: - contain multiple products and quantities; - have multiple financial transactions. A preorder authorization is considered an order. for these metrics to appear in the MVT Campaign portal , you must set up at least two candidates with the same Link Audience. You must also click the Control candidate button for the candidate you would like to be the control for the A/B Test.

Step 4: Open an MVT Campaign

  1. In the main menu, select Tools > MVT Campaigns. The MVT Campaigns search opens.
  2. Search for the campaign you want to view.
  3. Select the campaign line item in the search results.
  4. Click Open. The campaign opens.

 


Tip

When you are optimizing the checkout process and performing multivariate testing, take note of several best practices for achieving the best test results. See Best Practice: Test with MVT Campaigns.

Tip

An important best practice is to use a display URL on your website with an underlying destination URL. See Best Practice: Use Display URLs.