• Ashish Jose

How Much Should You Invest in Multivariate Testing?

Updated: Jun 9

One of the most crucial questions that troubles most of the digital marketers when it comes to Multivariate Testing is "How much will it cost to run an MVT experiment?". There is a common assumption among marketers that running a large number of creatives might cost them millions. However, the same is not true at all times. MVT can be used frequently and that too inexpensively; you just need to figure out how to build the experiments and the possible results that can be expected from them.

Understanding MVT experiments is certainly not a hard nut to crack, but to structure and run effective experiments, marketers need to utilize their creative and campaign assets intelligently. The ultimate goal of these experiments should be to find high performing creatives and draw learning about the creative components that clicked with your audiences such as colors, images,copy, graphics, etc.

Calculating the ad spend cost required for your first MVT is the first step in the process. Finding the overall cost will depend on the average cost per result (depending the industry KPI used) and the amount of creatives that you want to run for a particular campaign.

How to Evaluate Cost Per Result (CPR)? Finding the CPR would be easier if you already have previous campaign data. In case, if you're skeptical, you can take the average CPR of previous campaigns that you might have run with exactly the same KPI and use those numbers as the base to initiate your experiment. It's important to consider that your cost per result is dependent on what results you want to achieve and the purchases will cost more than leads which eventually cost more than clicks.

How to Evaluate Number of Creatives? Another aspect that you need to analyze is the number of creatives that you're going to make. Here are the three creative variables that need to be considered for your experiment:

  1. Creative Design

  2. Image

  3. Text Description

Creative Design refers to the high-level layout in which your texts, images, logos, graphics, etc. are displayed in the creative. It would be best if you use the most performing creative as a framework for all your creative designs, as you can always enhance it with the existing creative assets for future use. Once you are sure about the designs, apply multiple combinations images and text description to build your multivariate matrix. Make sure that every design, image, and text variant is combined with every other creative variable.

Consider the creative assets that you have build for running the MVT experiment with the following breakdown:

  • Creative Designs - 2

  • Images - 2

  • Text- 5

Now with 2 creative designs, 2 images, and 5 text descriptions, you can make 20 ads (2 x 2 x 5) that can be tested. In order to run those creatives for a particular set of audience at a $25 CPR, you'll require a budget of $500 (20 ads x $25 per creative). If that's not the amount that you are willing to spend, you should spend around 75% of the CPR on each creative, which would result in a total ad spend of $360 (20 ads x $18 per creative) for this test.

Concluding the experiment

Once you acquire the basic knowledge of building your MVT test structure, you can use the knowledge to figure out the expected results. If you plan to spend $500 on creatives, you can expect around 20 purchases (with the CPR being accurate).  On the other hand, if you opt for the $360 experiment, the purchases should be around 18.

Ideally, you will want to ensure that at least 15 purchases are made for the data to be relevant. However, even if you are able to crack smaller number of purchases, you will get impressive data if you acquired strong positive or negative user reactions.


©- 2019 Pyxis All rights reserved