Manage MVT campaigns
MVT campaigns are used to perform multivariate testing, as well as to route customers by region, language, browser, or operating system without performing testing.
Different versions of the checkout process, called candidates, are set up. When a customer begins the checkout process, one of the candidates is randomly assigned if the candidates have the same link audience.
A candidate ranking indicator displays for active, eligible candidates and shows you when a candidate becomes statistically significant and should be analyzed.
You can create MVT campaigns in two ways:
Create an MVT campaign
Create an MVT campaign in our web admin tool
-
Go to Optimize > MVT Campaigns.
-
Click Add MVT Campaign and enter a unique name for the campaign.
-
In the Display URL field, enter a unique identifier to be added to the end of the display URL. The first part of the URL is pre-filled. This URL is used as the campaign URL and is visible to customers.
-
Click the Copy
icon to copy the URL and paste it on your website. The Active checkbox defines whether this campaign is actively used for testing.
-
Add candidates to the campaign. Enter information in the following sections:
Test tab
The Test tab is used to configure an MVT campaign used for multivariate testing. To add a test to the MVT campaign, do the following:
-
Click Test.
-
Add a candidate to the MVT campaign. You can add as many candidates as necessary for the campaign. These candidates are randomly selected for customers during testing when the display URL is clicked. To add an MVT candidate, complete the following:
In the Candidate Name field, enter an identifier for the candidate that uniquely identifies what this candidate represents. This identifier is also used for reporting in the BI. Do not change the candidate name after you created the candidate. Each candidate gets a unique ID. It might happen that a name change results in two candidates with one ID.
The Active checkbox defines whether or not this candidate is actively used for testing.
In the Destination URL field, create a destination URL. This is the URL for this candidate and is not visible to customers.
noteIn the Candidates field, you can filter candidates by status, type, region, language, and date created.
tipTo remove or replace a parameter within the Destination URL of an MVT campaign, click Parameter Search and search for parameters by
<parametername>=<value>
or<parametername>
.You can determine how often your candidates are used in the campaign using the Candidate Weight section. If you give one of your candidates a higher percentage, this means that the candidate will be assigned to more customers.
By default, the Candidate Weight equally distributes the percentage in which particular candidates are assigned. If you want to distribute a candidate less than the other candidates, drag the toggle to the appropriate weight.
ExampleA fictional company called Shieldware creates an MVT campaign with three candidates. Candidate A is to be assigned 50% of the time, Candidate B is to be assigned 35% of the time, and Candidate C is to be assigned 15% of the time. To achieve this, they give the candidates the following weights:
Candidate name Candidate weight Candidate A 50 Candidate B 35 Candidate C 15 If Shieldware ever adds/deactivates a candidate, then they must adjust the weights of the remaining Active candidates and ensure that their combined weight reflects the intended traffic volume for each candidate.
noteIn the Control candidate field you can determine the candidate you would like to be the control for the A/B Test.
Define your audience for the candidate in the Audience Settings section. Select from the following options:
ImportantBy default, customers are always routed to the very first candidate assigned to them, even if they return to the checkout process using the same computer in a different situation (for example, they access the checkout process from different countries). Therefore, if you want to give customers a fresh candidate every time they access the checkout process, you must use the Reset MVT Cookie Parameter (
&resetMvtCandidate
). For more information, see Cookie Settings.noteWhen a URL is submitted that lacks a particular option (for example, Browser), and you have created four MVT candidates (three candidates with a particular option and one candidate without it), then only the candidate without the option will be considered.
Option Description Region Define a specific region for the candidate. This can be used to test different designs or configurations in different regions. To add a new region, click the Add new button.
ExampleCandidate A is displayed in Germany.
Candidate B is displayed in Switzerland.tipTo ensure the rest of the world has access to the standard checkout process, add another candidate and specify a region that covers all other regions not included in your region-specific candidates.
noteThe Region option has a higher priority than the Language option. Therefore, if you defined two candidates as follows:
- Candidate A: region = US
- Candidate B: language = Spanish
then a customer who accesses the checkout process with Spanish language settings and an IP address from the US will be given Candidate A.
Language Define a specific language for the candidate. The language is selected based on the customer's default browser language.
Browser Define a specific browser for the candidate.
Operating system Define a specific operating system for the candidate.
Parameter name Define an additional parameter for the candidate (for example,
audience=google
).ExampleThe fictional company, Shieldware, wants to conduct a single MVT campaign, but they want to test different elements on different types of customers:
- For all customers who initially access their checkout pages through the Shieldware website, they want to conduct an A/B test evaluating the success of the 2016 checkout-page layout compared to the 2017 layout.
- For all customers who initially access their checkout pages through marketing emails, they want to conduct an A/B test evaluating the success of pre-filled customer details compared to blank customer details.
- For all customers who initially access their checkout pages through a search engine, they do not want to conduct an A/B test. To accomplish this, Shieldware creates an MVT campaign named “Traffic” that contains five candidates. They name and define the candidates as follows:
Candidate name Definition Audience 1A Customers who initially access the checkout pages through the Shieldware website and receive the 2016 checkout-page layout. Audience 1B Customers who initially access the checkout pages through the Shieldware website and receive the 2017 checkout-page layout. Audience 2A Customers who initially access the checkout pages through a link in a marketing email and receive pre-filled customer details. Audience 2B Customers who initially access the checkout pages through a link in a marketing email and receive blank customer details. Audience 3 Customers who initially access the checkout pages through a link in a search engine and receive no test checkout pages. They insert the following values in the Parameter name field for each candidate:
Candidate name Parameter name Audience 1A shieldware
Audience 1B shieldware
Audience 2A email
Audience 2B email
Audience 3 search
When customers access Shieldware’s checkout pages for the first time, these parameters will route them to the appropriate candidates. For example, if a customer initially accesses the checkout pages through the Shieldware website, they will be randomly assigned to Audience 1A or Audience 1B.
https://www.shieldware.com/864/purl-anti?audience=shieldware
On the other hand, if a customer initially accesses the checkout pages through a marketing email, they will be randomly assigned to Audience 2A or Audience 2B.
https://www.shieldware.com/864/purl-anti?audience=email
tipIf you want to use the Parameter Name field to route customers to particular candidates, you must use the same values as for your traffic sources. For additional information, contact Client Experience.
tipTo ensure that MVT filtering is functioning properly, you should also add an additional or fallback candidate with no value in the Parameter Name field.
HTTP referrer pattern Define and HTTP referrer pattern for the candidate.
tipThe pattern must be written as a regular expression.
ExampleThe fictional company, Shieldware, has links to the checkout process on its landing page, its home page, and one of its content pages. They want to test different elements on different customers, depending on through which web page they accessed the checkout process.
To accomplish this, Shieldware creates an MVT campaign named “Referrer” that contains three candidates. They name the candidates as follows and define one HTTP referrer pattern for each one:Candidate name HTTP referrer pattern Audience 1 https://(.*?)/landing
Audience 2 https://(.*?)/home
Audience 3 https://(.*?)/content
When customers are forwarded to Shieldware’s checkout pages through one of these web pages, they will be routed to the appropriate candidate.
Enter detailed notes about the candidate's purpose.
tipAs the test progresses, use the notes field to keep track of the testing results, such as why a candidate was successful or not successful. That way you can keep a history of changes.
-
To add another candidate, click Add Candidate. Another row displays with the new candidate. Repeat steps 1-4.
-
Click Save and Close.
- Candidate A: region = US
Routings tab
The Routing tab is used to route customers by region, language, browser, or operating system without performing testing. To add Routing to the MVT campaign, do the following:
-
Click Add Routing.
-
Add a candidate to the campaign. To add an MVT candidate, complete the following:
In the Candidate Name field, enter an identifier for the candidate that uniquely identifies what this candidate represents.
The Active checkbox defines whether or not this candidate is actively used for routing.
In the Destination URL field, create a destination URL. This is the URL for this candidate and is not visible to customers.
noteIn the Candidates field, you can filter candidates by status, type, region, language, and date created.
tipTo remove or replace a parameter within the Destination URL of an MVT campaign, click Parameter Search and search for parameters by
<parametername>=<value>
or<parametername>
.Select from the following options:
Option Description Parameter name Define an additional parameter for the candidate (for example,
audience=google
).ExampleThe fictional company, Shieldware, wants to conduct a single MVT campaign, but they want to test different elements on different types of customers:
- For all customers who initially access their checkout pages through the Shieldware website, they want to conduct an A/B test evaluating the success of the 2016 checkout-page layout compared to the 2017 layout.
- For all customers who initially access their checkout pages through marketing emails, they want to conduct an A/B test evaluating the success of pre-filled customer details compared to blank customer details.
- For all customers who initially access their checkout pages through a search engine, they do not want to conduct an A/B test. To accomplish this, Shieldware creates an MVT campaign named “Traffic” that contains five candidates. They name and define the candidates as follows:
Candidate name Definition Audience 1A Customers who initially access the checkout pages through the Shieldware website and receive the 2016 checkout-page layout. Audience 1B Customers who initially access the checkout pages through the Shieldware website and receive the 2017 checkout-page layout. Audience 2A Customers who initially access the checkout pages through a link in a marketing email and receive pre-filled customer details. Audience 2B Customers who initially access the checkout pages through a link in a marketing email and receive blank customer details. Audience 3 Customers who initially access the checkout pages through a link in a search engine and receive no test checkout pages. They insert the following values in the Parameter name field for each candidate:
Candidate name Parameter name Audience 1A shieldware
Audience 1B shieldware
Audience 2A email
Audience 2B email
Audience 3 search
When customers access Shieldware’s checkout pages for the first time, these parameters will route them to the appropriate candidates. For example, if a customer initially accesses the checkout pages through the Shieldware website, they will be randomly assigned to Audience 1A or Audience 1B.
https://www.shieldware.com/864/purl-anti?audience=shieldware
On the other hand, if a customer initially accesses the checkout pages through a marketing email, they will be randomly assigned to Audience 2A or Audience 2B.
https://www.shieldware.com/864/purl-anti?audience=email
tipIf you want to use the Parameter Name field to route customers to particular candidates, you must use the same values as for your traffic sources. For additional information, contact Client Experience.
tipTo ensure that MVT filtering is functioning properly, you should also add an additional or fallback candidate with no value in the Parameter Name field.
HTTP referrer pattern Define and HTTP referrer pattern for the candidate.
tipThe pattern must be written as a regular expression.
ExampleThe fictional company, Shieldware, has links to the checkout process on its landing page, its home page, and one of its content pages. They want to test different elements on different customers, depending on through which web page they accessed the checkout process.
To accomplish this, Shieldware creates an MVT campaign named “Referrer” that contains three candidates. They name the candidates as follows and define one HTTP referrer pattern for each one:Candidate name HTTP referrer pattern Audience 1 https://(.*?)/landing
Audience 2 https://(.*?)/home
Audience 3 https://(.*?)/content
When customers are forwarded to Shieldware’s checkout pages through one of these web pages, they will be routed to the appropriate candidate.
Define your audience for the candidate in the Audience Settings section. Select from the following options:
ImportantBy default, customers are always routed to the very first candidate assigned to them, even if they return to the checkout process using the same computer in a different situation (for example, they access the checkout process from different countries). Therefore, if you want to give customers a fresh candidate every time they access the checkout process, you must use the Reset MVT Cookie Parameter (
&resetMvtCandidate
). For more information, see Cookie Settings.noteWhen a URL is submitted that lacks a particular option (for example, Browser), and you have created four MVT candidates (three candidates with a particular option and one candidate without it), then only the candidate without the option will be considered.
Option Description Region Define a specific region for the candidate. This can be used to test different designs or configurations in different regions. To add a new region, click the Add new button.
ExampleCandidate A is displayed in Germany.
Candidate B is displayed in Switzerland.tipTo ensure the rest of the world has access to the standard checkout process, add another candidate and specify a region that covers all other regions not included in your region-specific candidates.
noteThe Region option has a higher priority than the Language option. Therefore, if you defined two candidates as follows:
- Candidate A: region = US
- Candidate B: language = Spanish
then a customer who accesses the checkout process with Spanish language settings and an IP address from the US will be given Candidate A.
Language Define a specific language for the candidate. The language is selected based on the customer's default browser language.
Browser Define a specific browser for the candidate.
Operating system Define a specific operating system for the candidate.
- Enter detailed notes about the purpose of the candidate.
- Click Save and Close.
For detailed instructions on how to create an MVT Campaign in CA, see MVT Campaigns.
Monitor an MVT campaign
Monitor an MVT campaign in our web admin tool
There are two ways to monitor your campaigns:
Rankings
The V on the right-hand side of the candidate name is used to rank the success of a candidate during testing. It provides an indication of when a campaign is statistically significant enough for you to begin analyzing the test results.
The ranking is not intended to provide sufficient information about which candidate performs best. It is only a relative result that needs more evaluation to generate insight for your business.
Candidates are ranked according to the following rules:
- Ranking is eligible for active candidates testing against one another, which means two or more candidates with the same Link Audience settings.
If you have three candidates and only two have the same Link Audience settings, then only those two candidates are ranked.
If you have four candidates and two have the same Link Audience settings, and the other two have a different set of Link Audience settings, both groups of two will be ranked, but in two separate tests.
Specific example
You have specified a region for Candidate A (for example, Europe), and then you also need to specify a region for Candidate B (for example, USA). If you also want to test an operating system, you need to set up one more candidate with Phone as operating system and World as region.
- Ranking is based on revenue per session. A candidate with the ranking indicator V shows that this campaign worked considerably better than the other candidate. However, the confidence level is significant. Contact Client Experience to help you evaluate the results in an Excel sheet. Client Experience can assist you with creating forms to calculate the confidence level.
Always use new candidates when you run tests. If you use an existing candidate to compare to a newly specified candidate, the users per session of the already existing candidate is taken into account.
Metrics
The metrics under the Candidate Name can help you determine which candidates to activate permanently. The following explains how to interpret them:
Metric | Description |
---|---|
Orders/Sessions | Orders are the number of transactions completed and paid for using the checkout pages pertaining to this candidate. Sessions are the number of visits to the checkout pages pertaining to this candidate. |
CR | Conversion rate (CR) is the number of orders divided by the number of sessions. *(CR = Orders / Sessions) |
Uplift | Uplift is the new conversion rate divided by the control conversion rate minus 1. *(Uplift = (New CR / Control CR) - 1) |
Significance | Significance is the probability that your A/B test yielded trustworthy results. If you have a statistical significance of 90-100%, you can have a high level of confidence in the validity of your results. |
In order for these metrics to appear in the MVT Campaign section, you must set up at least two candidates with the same Link Audience. You must also determine the Control candidate for the candidate you would like to be the control for the A/B Test.
For information on how to monitor an MVT Campaign in CA, see MVT Campaigns.