Manage MVT campaigns

MVT campaigns are used to perform multivariate testing, as well as to route customers by region, language, browser, or operating system without performing testing.

Different versions of the checkout process, called candidates, are set up. When a customer begins the checkout process, one of the candidates is randomly assigned if the candidates have the same link audience. A candidate ranking indicator displays for active, eligible candidates and shows you when a candidate becomes statistically significant and should be analyzed.

Create an MVT campaign

1. Go to Optimize > MVT Campaigns.

2. Click Add MVT Campaign and enter a unique name for the campaign.

3. In the Display URL field, enter a unique identifier to be added to the end of the display URL. The first part of the URL is pre-filled. This URL is used as the campaign URL and is visible to customers.

4. Click the copy icon to copy the URL and paste it on your website. The Active checkbox defines whether this campaign is actively used for testing.

5. Add candidates to the campaign. Enter information in the following sections:

Test

Test

The Test tab is used to configure an MVT campaign used for multivariate testing. To add a test to the MVT campaign, do the following:

1. Click Test.
2. Add a candidate to the MVT campaign. You can add as many candidates as necessary for the campaign. These candidates are randomly selected for customers during testing when the display URL is clicked. To add an MVT candidate, complete the following:

Enter a candidate identifier

In the Candidate Name field, enter an identifier for the candidate that uniquely identifies what this candidate represents. This identifier is also used for reporting in the BI. Do not change the candidate name after you created the candidate. Each candidate gets a unique ID. It might happen that a name change results in two candidates with one ID.

The Active checkbox defines whether or not this candidate is actively used for testing.

Create a destination URL

In the Destination URL field, create a destination URL. This is the URL for this candidate and is not visible to customers.

In the Candidates field, you can filter candidates by status, type, region, language, and date created.

tip

To remove or replace a parameter within the Destination URL of an MVT campaign, click Parameter Search and search for parameters by <parametername>=<value> or <parametername>.

Assign traffic volume to candidates

Assign traffic volume to candidates to control how often each candidate is shown during the campaign.

Candidate weight behavior

Use the Candidate Weight section to determine how often each candidate is used in the campaign. A candidate with a higher percentage is assigned to more customers.

By default, Candidate Weight distributes traffic equally among candidates. To assign less traffic to one candidate than to the others, drag the toggle to the appropriate weight.

Example

A fictional company called Shieldware creates an MVT campaign with three candidates. Candidate A is to be assigned 50% of the time, Candidate B is to be assigned 35% of the time, and Candidate C is to be assigned 15% of the time. To achieve this, they give the candidates the following weights:

Candidate name	Candidate weight
Candidate A	50
Candidate B	35
Candidate C	15

If Shieldware ever adds/deactivates a candidate, then they must adjust the weights of the remaining Active candidates and ensure that their combined weight reflects the intended traffic volume for each candidate.

note

In the Control candidate field you can determine the candidate you would like to be the control for the A/B Test.

Specify the link audience

Specify the link audience to define which customers are routed to this candidate based on criteria such as region, language, browser, operating system, URL parameters, or referrer patterns.

Audience settings

Define your audience for the candidate in the Audience Settings section. Select from the following options:

View audience settings and routing examples

Important

By default, customers are always routed to the very first candidate assigned to them, even if they return to the checkout process using the same computer in a different situation (for example, they access the checkout process from different countries). Therefore, if you want to give customers a fresh candidate every time they access the checkout process, you must use the Reset MVT Cookie Parameter (&resetMvtCandidate). For more information, see Cookie Settings.

note

When a URL is submitted that lacks a particular option (for example, Browser), and you have created four MVT candidates (three candidates with a particular option and one candidate without it), then only the candidate without the option will be considered.

Option	Description
Region	Define a specific region for the candidate. This can be used to test different designs or configurations in different regions. To add a new region, click the Add new button. Example Candidate A is displayed in Germany. Candidate B is displayed in Switzerland. tip To ensure the rest of the world has access to the standard checkout process, add another candidate and specify a region that covers all other regions not included in your region-specific candidates. note The Region option has a higher priority than the Language option. Therefore, if you defined two candidates as follows: Candidate A: region = US Candidate B: language = Spanish then a customer who accesses the checkout process with Spanish language settings and an IP address from the US will be given Candidate A.
Language	Define a specific language for the candidate. The language is selected based on the customer's default browser language.
Browser	Define a specific browser for the candidate.
Operating system	Define a specific operating system for the candidate.
Parameter name	Define an additional parameter for the candidate (for example, audience=google). Example The fictional company, Shieldware, wants to conduct a single MVT campaign, but they want to test different elements on different types of customers: For all customers who initially access their checkout pages through the Shieldware website, they want to conduct an A/B test evaluating the success of the 2016 checkout-page layout compared to the 2017 layout. For all customers who initially access their checkout pages through marketing emails, they want to conduct an A/B test evaluating the success of pre-filled customer details compared to blank customer details. For all customers who initially access their checkout pages through a search engine, they do not want to conduct an A/B test. To accomplish this, Shieldware creates an MVT campaign named "Traffic" that contains five candidates. They name and define the candidates as follows: Candidate name Definition Audience 1A Customers who initially access the checkout pages through the Shieldware website and receive the 2016 checkout-page layout. Audience 1B Customers who initially access the checkout pages through the Shieldware website and receive the 2017 checkout-page layout. Audience 2A Customers who initially access the checkout pages through a link in a marketing email and receive pre-filled customer details. Audience 2B Customers who initially access the checkout pages through a link in a marketing email and receive blank customer details. Audience 3 Customers who initially access the checkout pages through a link in a search engine and receive no test checkout pages. They insert the following values in the Parameter name field for each candidate: Candidate name Parameter name Audience 1A shieldware Audience 1B shieldware Audience 2A email Audience 2B email Audience 3 search When customers access Shieldware's checkout pages for the first time, these parameters will route them to the appropriate candidates. For example, if a customer initially accesses the checkout pages through the Shieldware website, they will be randomly assigned to Audience 1A or Audience 1B. https://www.shieldware.com/864/purl-anti?audience=shieldware On the other hand, if a customer initially accesses the checkout pages through a marketing email, they will be randomly assigned to Audience 2A or Audience 2B. https://www.shieldware.com/864/purl-anti?audience=email tip If you want to use the Parameter Name field to route customers to particular candidates, you must use the same values as for your traffic sources. For additional information, contact Client Experience. tip To ensure that MVT filtering is functioning properly, you should also add an additional or fallback candidate with no value in the Parameter Name field.
HTTP referrer pattern	Define and HTTP referrer pattern for the candidate. tip The pattern must be written as a regular expression. Example The fictional company, Shieldware, has links to the checkout process on its landing page, its home page, and one of its content pages. They want to test different elements on different customers, depending on through which web page they accessed the checkout process. To accomplish this, Shieldware creates an MVT campaign named "Referrer" that contains three candidates. They name the candidates as follows and define one HTTP referrer pattern for each one: Candidate name HTTP referrer pattern Audience 1 https://(.*?)/landing Audience 2 https://(.*?)/home Audience 3 https://(.*?)/content When customers are forwarded to Shieldware's checkout pages through one of these web pages, they will be routed to the appropriate candidate.

Enter notes

Enter detailed notes about the candidate's purpose.

tip

As the test progresses, use the notes field to keep track of the testing results, such as why a candidate was successful or not successful. That way you can keep a history of changes.

Add more candidates

1. To add another candidate, click Add Candidate. Another row displays with the new candidate. Repeat steps 1-4.

2. Click Save and Close.

Routing

Routing

The Routing tab is used to route customers by region, language, browser, or operating system without performing testing. To add Routing to the MVT campaign, do the following:

1. Click Add Routing.

2. Add a candidate to the campaign. To add an MVT candidate, complete the following:

   Enter a candidate identifier

   In the Candidate Name field, enter an identifier for the candidate that uniquely identifies what this candidate represents.

The Active checkbox defines whether or not this candidate is actively used for routing.

Create a destination URL

In the Destination URL field, create a destination URL. This is the URL for this candidate and is not visible to customers.

note

In the Candidates field, you can filter candidates by status, type, region, language, and date created.

tip

To remove or replace a parameter within the Destination URL of an MVT campaign, click Parameter Search and search for parameters by <parametername>=<value> or <parametername>.

Define characteristics for candidates

Select from the following options:

View candidate characteristics

Option	Description
Parameter name	Define an additional parameter for the candidate (for example, audience=google). Example The fictional company, Shieldware, wants to conduct a single MVT campaign, but they want to test different elements on different types of customers: For all customers who initially access their checkout pages through the Shieldware website, they want to conduct an A/B test evaluating the success of the 2016 checkout-page layout compared to the 2017 layout. For all customers who initially access their checkout pages through marketing emails, they want to conduct an A/B test evaluating the success of pre-filled customer details compared to blank customer details. For all customers who initially access their checkout pages through a search engine, they do not want to conduct an A/B test. To accomplish this, Shieldware creates an MVT campaign named "Traffic" that contains five candidates. They name and define the candidates as follows: Candidate name Definition Audience 1A Customers who initially access the checkout pages through the Shieldware website and receive the 2016 checkout-page layout. Audience 1B Customers who initially access the checkout pages through the Shieldware website and receive the 2017 checkout-page layout. Audience 2A Customers who initially access the checkout pages through a link in a marketing email and receive pre-filled customer details. Audience 2B Customers who initially access the checkout pages through a link in a marketing email and receive blank customer details. Audience 3 Customers who initially access the checkout pages through a link in a search engine and receive no test checkout pages. They insert the following values in the Parameter name field for each candidate: Candidate name Parameter name Audience 1A shieldware Audience 1B shieldware Audience 2A email Audience 2B email Audience 3 search When customers access Shieldware's checkout pages for the first time, these parameters will route them to the appropriate candidates. For example, if a customer initially accesses the checkout pages through the Shieldware website, they will be randomly assigned to Audience 1A or Audience 1B. https://www.shieldware.com/864/purl-anti?audience=shieldware On the other hand, if a customer initially accesses the checkout pages through a marketing email, they will be randomly assigned to Audience 2A or Audience 2B. https://www.shieldware.com/864/purl-anti?audience=email tip If you want to use the Parameter Name field to route customers to particular candidates, you must use the same values as for your traffic sources. For additional information, contact Client Experience. tip To ensure that MVT filtering is functioning properly, you should also add an additional or fallback candidate with no value in the Parameter Name field.
HTTP referrer pattern	Define and HTTP referrer pattern for the candidate. tip The pattern must be written as a regular expression. Example The fictional company, Shieldware, has links to the checkout process on its landing page, its home page, and one of its content pages. They want to test different elements on different customers, depending on through which web page they accessed the checkout process. To accomplish this, Shieldware creates an MVT campaign named "Referrer" that contains three candidates. They name the candidates as follows and define one HTTP referrer pattern for each one: Candidate name HTTP referrer pattern Audience 1 https://(.*?)/landing Audience 2 https://(.*?)/home Audience 3 https://(.*?)/content When customers are forwarded to Shieldware's checkout pages through one of these web pages, they will be routed to the appropriate candidate.

Specify the link audience

Define your audience for the candidate in the Audience Settings section. Select from the following options:

View routing audience settings

Important

By default, customers are always routed to the very first candidate assigned to them, even if they return to the checkout process using the same computer in a different situation (for example, they access the checkout process from different countries). Therefore, if you want to give customers a fresh candidate every time they access the checkout process, you must use the Reset MVT Cookie Parameter (&resetMvtCandidate). For more information, see Cookie Settings.

note

When a URL is submitted that lacks a particular option (for example, Browser), and you have created four MVT candidates (three candidates with a particular option and one candidate without it), then only the candidate without the option will be considered.

Option	Description
Region	Define a specific region for the candidate. This can be used to test different designs or configurations in different regions. To add a new region, click the Add new button. Example Candidate A is displayed in Germany. Candidate B is displayed in Switzerland. tip To ensure the rest of the world has access to the standard checkout process, add another candidate and specify a region that covers all other regions not included in your region-specific candidates. note The Region option has a higher priority than the Language option. Therefore, if you defined two candidates as follows: Candidate A: region = US Candidate B: language = Spanish then a customer who accesses the checkout process with Spanish language settings and an IP address from the US will be given Candidate A.
Language	Define a specific language for the candidate. The language is selected based on the customer's default browser language.
Browser	Define a specific browser for the candidate.
Operating system	Define a specific operating system for the candidate.

Enter notes

1. Enter detailed notes about the purpose of the candidate.
2. Click Save and Close.

note

For detailed instructions on how to create an MVT Campaign in CA, see MVT Campaigns.

Monitor an MVT campaign

Rankings

The V on the right-hand side of the candidate name is used to rank the success of a candidate during testing. It provides an indication of when a campaign is statistically significant enough for you to begin analyzing the test results.

note

The ranking is not intended to provide sufficient information about which candidate performs best. It is only a relative result that needs more evaluation to generate insight for your business.

Candidates are ranked according to the following rules:

• Ranking is eligible for active candidates testing against one another, which means two or more candidates with the same Link Audience settings.

Example

If you have three candidates and only two have the same Link Audience settings, then only those two candidates are ranked.

If you have four candidates and two have the same Link Audience settings, and the other two have a different set of Link Audience settings, both groups of two will be ranked, but in two separate tests.

Specific example

You have specified a region for Candidate A (for example, Europe), and then you also need to specify a region for Candidate B (for example, USA). If you also want to test an operating system, you need to set up one more candidate with Phone as operating system and World as region.

• Ranking is based on revenue per session. A candidate with the ranking indicator V shows that this campaign worked considerably better than the other candidate. However, the confidence level is significant. Contact Client Experience to help you evaluate the results in an Excel sheet. Client Experience can assist you with creating forms to calculate the confidence level.

note

Always use new candidates when you run tests. If you use an existing candidate to compare to a newly specified candidate, the users per session of the already existing candidate is taken into account.

Metrics

The metrics under the Candidate Name can help you determine which candidates to activate permanently. Use the following information to interpret them:

Metric	Description
Orders/Sessions	Orders are the number of transactions completed and paid for using the checkout pages pertaining to this candidate. Sessions are the number of visits to the checkout pages pertaining to this candidate.
CR	Conversion rate (CR) is the number of orders divided by the number of sessions. CR = Orders / Sessions
Uplift	Uplift is the new conversion rate divided by the control conversion rate minus 1. Uplift = (New CR / Control CR) - 1
Significance	Significance is the probability that your A/B test yielded trustworthy results. If you have a statistical significance of 90-100%, you can have a high level of confidence in the validity of your results.

Important

In order for these metrics to appear in the MVT Campaign section, you must set up at least two candidates with the same Link Audience. You must also determine the Control candidate for the candidate you would like to be the control for the A/B Test.

note

For information on how to monitor an MVT Campaign in CA, see MVT Campaigns.

Did you find this doc useful?