How To Think About Experiment Resolution
.png)
Ask a professional artist who’s spent years studying sight-size technique in Florence, how long a cast drawing takes, and they’ll tell you, “probably 2–3 weeks?”
Ask a student doing a portfolio prep class at Central Saint Martins the same question, and the answer will likely be “a day or an afternoon?”
I’m not talking about the Dunning–Kruger effect here, but something entirely different.
Imagine the two images. What is the main difference?
I’d put money on it being immediately obvious to distinguish between the artists based on the differing levels of detail achieved in a day versus three weeks.
But what’s this got to do with A/B testing?
Well, A/B test durations function like drawing: the resolution (or detail) of the drawing is directly proportional to the duration of an A/B test.
Resolution & A/B Test Durations
Businesses don’t often need the same level of statistical rigor as medical experiments, where test outcomes can have life or death consequences. But we do have to answer to stakeholders, and our test results can inform financial investments. So it’s important to understand the resolution you need to achieve so that your test result satisfies the business needs.
Using our art analogy, imagine two different images.
The image on the left has mostly a high value (or light) area, and a low value (or dark) area down the left-hand side. In the dark areas, we can’t see any fiddly details; it’s just dark. In this case, you don’t need a high-resolution image to understand it.
The image on the right has no such simplicity: areas move in and out of high, mid, and low values. The differences between the light and dark in this image are much more nuanced and harder to distinguish. To observe all of the details in this image, you need a much higher resolution.
So, how long do you think it takes to reach “accurate resolution” in each case?
For our “simple” image, we get to the essential answers — such as where is it light and dark — by day three. We keep going to day five to ensure completeness, but we are only getting incremental gains.
For the “detailed” image, we only really get clarity by day six, and we could have kept going to ensure we had even more detail.
How well did our sample replicate the underlying distribution? Very well in this case, but it took a lot more time and effort, and uncertainty.
This example echoes the questions we need to ask when deciding on A/B test duration. To what level do we need our sample data to approximate the underlying distribution?
Digging Deeper into A/B Test Durations
So, A/B test durations function just like drawing, with resolution directly proportional to test duration:
Lots of data (like our high-resolution image) = Long test duration
Less data (like our simple image) = Short test duration
Or put another way, the size of the Minimum Detectable Effect (MDE) is inversely proportional to test duration:
- Large MDE = Short Duration
- Small MDE = Long Duration
MDE also impacts sample size (which impacts test duration), and this is also inversely proportional:
- Large MDE = Small Sample Size
- Small MDE = Large Sample Size
Putting that all together, you end up with the following:
This is why we need to answer the following questions when working out a test duration:
- How much data do we need to collect to get the desired resolution?
- What level of detail is needed to observe the effect?
We need to think about sample size and test duration as “zeroing in” on an ever smaller target, which requires more and more data to uncover. To find a tiny target, you need less uncertainty, which means more data.
But there’s more to working out a test duration. In reality, many additional factors impact the test duration decision:
- Your level of experience and statistical knowledge. The more experienced you are, the more you desire highly accurate results. We tend to “feel bad” if we don’t collect enough data to be able to defend the test results. But this needs to be balanced with the business risk appetite and commercial needs.
- The type of test. The more complex and fiddly the features you test, the more time, effort, and investment will go into them. The more riding on a test, the higher we want the probability (significance) of the results being true. Other test design factors, like the number of variants or statistical models you are using (e.g., multivariate or sequential test), will impact test duration as well as stopping rules.
- External factors. Seasonality, marketing campaigns, and sales cycles can all impact the results of a test and, therefore, are also factors to consider when setting the test duration.
How to Calculate Test Duration
Test duration is based on how long it takes for the sample size to be reached. The resolution, along with the other factors mentioned above, will impact the sample size you need to match your criteria. To calculate the sample size needed, use one of the many online A/B test calculators to help you with the maths. The calculation will use the volume of your site traffic to estimate how long it will take to achieve the sample size. A low-traffic site will take longer.
But don’t follow the A/B test calculators blindly, as they won’t take all of the factors mentioned above into account — for example, marketing campaigns planned during the test period.
For more help with calculating test durations, Watch the full talk I gave about analytics ans statistics for AB Testing here Eclipse Talks: Return to Basics — Analytics & A/B Testing, where we have also provided all the resources mentioned in the presentation.
If A/B test durations and statistics are still troubling you, feel free to get in touch with me about a custom in-house statistics workshop.
.png)
