Skip to main content

Manage MVT campaigns

MVT campaigns are used to perform multivariate testing, as well as to route customers by region, language, browser, or operating system without performing testing.

Different versions of the checkout process, called candidates, are set up. When a customer begins the checkout process, one of the candidates is randomly assigned if the candidates have the same link audience.

A candidate ranking indicator displays for active, eligible candidates and shows you when a candidate becomes statistically significant and should be analyzed.

You can create MVT campaigns in two ways:

Create an MVT campaign

Create an MVT campaign in our web admin tool

  1. Go to Optimize > MVT Campaigns.

  2. Click Add MVT Campaign and enter a unique name for the campaign.

  3. In the Display URL field, enter a unique identifier to be added to the end of the display URL. The first part of the URL is pre-filled. This URL is used as the campaign URL and is visible to customers.

  4. Click the Copy Checkout flows icon to copy the URL and paste it on your website. The Active checkbox defines whether this campaign is actively used for testing.

  5. Add candidates to the campaign. Enter information in the following sections:

Test tab

The Test tab is used to configure an MVT campaign used for multivariate testing. To add a test to the MVT campaign, do the following:

  1. Click Test.

  2. Add a candidate to the MVT campaign. You can add as many candidates as necessary for the campaign. These candidates are randomly selected for customers during testing when the display URL is clicked. To add an MVT candidate, complete the following:

    In the Candidate Name field, enter an identifier for the candidate that uniquely identifies what this candidate represents. This identifier is also used for reporting in the BI. Do not change the candidate name after you created the candidate. Each candidate gets a unique ID. It might happen that a name change results in two candidates with one ID.

    The Active checkbox defines whether or not this candidate is actively used for testing.

    In the Destination URL field, create a destination URL. This is the URL for this candidate and is not visible to customers.

    note

    In the Candidates field, you can filter candidates by status, type, region, language, and date created.

    tip

    To remove or replace a parameter within the Destination URL of an MVT campaign, click Parameter Search and search for parameters by <parametername>=<value> or <parametername>.

    You can determine how often your candidates are used in the campaign using the Candidate Weight section. If you give one of your candidates a higher percentage, this means that the candidate will be assigned to more customers.

    By default, the Candidate Weight equally distributes the percentage in which particular candidates are assigned. If you want to distribute a candidate less than the other candidates, drag the toggle to the appropriate weight.

    Example

    A fictional company called Shieldware creates an MVT campaign with three candidates. Candidate A is to be assigned 50% of the time, Candidate B is to be assigned 35% of the time, and Candidate C is to be assigned 15% of the time. To achieve this, they give the candidates the following weights:

    Candidate nameCandidate weight
    Candidate A50
    Candidate B35
    Candidate C15

    If Shieldware ever adds/deactivates a candidate, then they must adjust the weights of the remaining Active candidates and ensure that their combined weight reflects the intended traffic volume for each candidate.

    note

    In the Control candidate field you can determine the candidate you would like to be the control for the A/B Test.

    Enter detailed notes about the candidate's purpose.

    tip

    As the test progresses, use the notes field to keep track of the testing results, such as why a candidate was successful or not successful. That way you can keep a history of changes.

    1. To add another candidate, click Add Candidate. Another row displays with the new candidate. Repeat steps 1-4.

    2. Click Save and Close.

Routings tab

The Routing tab is used to route customers by region, language, browser, or operating system without performing testing. To add Routing to the MVT campaign, do the following:

  1. Click Add Routing.

  2. Add a candidate to the campaign. To add an MVT candidate, complete the following:

    In the Candidate Name field, enter an identifier for the candidate that uniquely identifies what this candidate represents.

    The Active checkbox defines whether or not this candidate is actively used for routing.

    In the Destination URL field, create a destination URL. This is the URL for this candidate and is not visible to customers.

    note

    In the Candidates field, you can filter candidates by status, type, region, language, and date created.

    tip

    To remove or replace a parameter within the Destination URL of an MVT campaign, click Parameter Search and search for parameters by <parametername>=<value> or <parametername>.

    Select from the following options:

    OptionDescription
    Parameter name

    Define an additional parameter for the candidate (for example, audience=google).

    Example

    The fictional company, Shieldware, wants to conduct a single MVT campaign, but they want to test different elements on different types of customers:

    • For all customers who initially access their checkout pages through the Shieldware website, they want to conduct an A/B test evaluating the success of the 2016 checkout-page layout compared to the 2017 layout.
    • For all customers who initially access their checkout pages through marketing emails, they want to conduct an A/B test evaluating the success of pre-filled customer details compared to blank customer details.
    • For all customers who initially access their checkout pages through a search engine, they do not want to conduct an A/B test. To accomplish this, Shieldware creates an MVT campaign named “Traffic” that contains five candidates. They name and define the candidates as follows:
    Candidate nameDefinition
    Audience 1ACustomers who initially access the checkout pages through the Shieldware website and receive the 2016 checkout-page layout.
    Audience 1BCustomers who initially access the checkout pages through the Shieldware website and receive the 2017 checkout-page layout.
    Audience 2ACustomers who initially access the checkout pages through a link in a marketing email and receive pre-filled customer details.
    Audience 2BCustomers who initially access the checkout pages through a link in a marketing email and receive blank customer details.
    Audience 3Customers who initially access the checkout pages through a link in a search engine and receive no test checkout pages.

    They insert the following values in the Parameter name field for each candidate:

    Candidate nameParameter name
    Audience 1Ashieldware
    Audience 1Bshieldware
    Audience 2Aemail
    Audience 2Bemail
    Audience 3search

    When customers access Shieldware’s checkout pages for the first time, these parameters will route them to the appropriate candidates. For example, if a customer initially accesses the checkout pages through the Shieldware website, they will be randomly assigned to Audience 1A or Audience 1B.
    https://www.shieldware.com/864/purl-anti?audience=shieldware
    On the other hand, if a customer initially accesses the checkout pages through a marketing email, they will be randomly assigned to Audience 2A or Audience 2B.
    https://www.shieldware.com/864/purl-anti?audience=email

    tip

    If you want to use the Parameter Name field to route customers to particular candidates, you must use the same values as for your traffic sources. For additional information, contact Client Experience.

    tip

    To ensure that MVT filtering is functioning properly, you should also add an additional or fallback candidate with no value in the Parameter Name field.

    HTTP referrer pattern

    Define and HTTP referrer pattern for the candidate.

    tip

    The pattern must be written as a regular expression.

    Example

    The fictional company, Shieldware, has links to the checkout process on its landing page, its home page, and one of its content pages. They want to test different elements on different customers, depending on through which web page they accessed the checkout process.
    To accomplish this, Shieldware creates an MVT campaign named “Referrer” that contains three candidates. They name the candidates as follows and define one HTTP referrer pattern for each one:

    Candidate nameHTTP referrer pattern
    Audience 1https://(.*?)/landing
    Audience 2https://(.*?)/home
    Audience 3https://(.*?)/content

    When customers are forwarded to Shieldware’s checkout pages through one of these web pages, they will be routed to the appropriate candidate.

    1. Enter detailed notes about the purpose of the candidate.
    2. Click Save and Close.
note

For detailed instructions on how to create an MVT Campaign in CA, see MVT Campaigns.

Monitor an MVT campaign

Monitor an MVT campaign in our web admin tool

There are two ways to monitor your campaigns:

Rankings

The V on the right-hand side of the candidate name is used to rank the success of a candidate during testing. It provides an indication of when a campaign is statistically significant enough for you to begin analyzing the test results.

note

The ranking is not intended to provide sufficient information about which candidate performs best. It is only a relative result that needs more evaluation to generate insight for your business.

Candidates are ranked according to the following rules:

  • Ranking is eligible for active candidates testing against one another, which means two or more candidates with the same Link Audience settings.
Example

If you have three candidates and only two have the same Link Audience settings, then only those two candidates are ranked.

If you have four candidates and two have the same Link Audience settings, and the other two have a different set of Link Audience settings, both groups of two will be ranked, but in two separate tests.

Specific example

You have specified a region for Candidate A (for example, Europe), and then you also need to specify a region for Candidate B (for example, USA). If you also want to test an operating system, you need to set up one more candidate with Phone as operating system and World as region.

  • Ranking is based on revenue per session. A candidate with the ranking indicator V shows that this campaign worked considerably better than the other candidate. However, the confidence level is significant. Contact Client Experience to help you evaluate the results in an Excel sheet. Client Experience can assist you with creating forms to calculate the confidence level.
note

Always use new candidates when you run tests. If you use an existing candidate to compare to a newly specified candidate, the users per session of the already existing candidate is taken into account.

Metrics

The metrics under the Candidate Name can help you determine which candidates to activate permanently. The following explains how to interpret them:

MetricDescription
Orders/SessionsOrders are the number of transactions completed and paid for using the checkout pages pertaining to this candidate.
Sessions are the number of visits to the checkout pages pertaining to this candidate.
CRConversion rate (CR) is the number of orders divided by the number of sessions.
*(CR = Orders / Sessions)
UpliftUplift is the new conversion rate divided by the control conversion rate minus 1.
*(Uplift = (New CR / Control CR) - 1)
SignificanceSignificance is the probability that your A/B test yielded trustworthy results. If you have a statistical significance of 90-100%, you can have a high level of confidence in the validity of your results.
Important

In order for these metrics to appear in the MVT Campaign section, you must set up at least two candidates with the same Link Audience. You must also determine the Control candidate for the candidate you would like to be the control for the A/B Test.

note

For information on how to monitor an MVT Campaign in CA, see MVT Campaigns.


Did you find this doc useful?