The Inclusion Mindset: There is No Such Thing as an Average User
A/B tests are the gold standard for evidence-based decisions, but they require reliable and valid data and test design.
Bad tests occur when you don’t consider the full picture — meaning a certain (potentially large) subset of your data and users aren’t taken into account. While you might be familiar with the following example, it’s the perfect example of the perils of excluding data.
During World War II, the US military conducted a detailed examination of the bullet holes on combat aircraft. Based on this, they decided to add reinforced armour to the areas where the plane received most damage.
But statistician Abraham Wald recommended they reinforce the areas with the fewest bullet holes. Why?
The military’s dataset only included planes that returned to base. The aircraft that sustained fatal damage were missing entirely. Abraham inferred that the planes that made it back sustained fewer critical attacks and were still able to fly. His insights were based on survivorship bias.
If you’re not accounting for the users you don’t see, you might be reinforcing the wrong parts of the plane. Similarly, the users of your website who either leave immediately or skip your site altogether are absent from your analytics — you don’t know what you don’t have evidence for.
The Average User
If you have a persona of your average customer, you might want to look away now.
Often, when teams develop hypotheses or design experiments, they design for the ‘average’ user or someone who reaches a particular stage in the experience, ignoring those who don’t reach or fit this average. But the average person isn’t the majority and may not even exist.
In 2024, a quarter of the UK population had a disability. This figure doesn’t include people with temporary or situational impairments. That’s a huge proportion of people not to consider when designing experiments.
Proactively accommodating those outside the “average” makes you a better experimenter and makes life easier for people with access needs. You start to think more about real individuals rather than a synthesised, ‘average’ person, and ultimately create a better website experience for all users, including speed, SEO and AI.
Distinguishing Differences
My mother was disabled since I was seven years old. This meant I was constantly looking for alternatives that better suited her needs.
When you look after someone with access needs, you:
- Choose the best option among several sub-optimal ones.
- Actively create or seek alternatives.
- Examine permutations and interaction effects.
The same can (and should) be applied to experiment design.
To show what this looks like in reality, here are two examples.
Example: A Day in The Great British Outdoors
Imagine you are trying to pick the right outerwear for a trip you’re taking. If you’ve spent any time in the UK, you’ll know that the forecast often changes. It can rain hard and suddenly, and not having a coat or umbrella is bad (it’s often that fine rain that soaks you right through!).
But if you take a coat and umbrella and it’s a warm, dry day, then you’re carrying heavy stuff around for no reason. If we set this out in a matrix, we can determine which is the best option. In this scenario, taking either an umbrella or a coat is the best overall option.
Let’s make this more complicated. It might not only be rainy, but sometimes it can be rainy and cold. With this additional variable, having a coat becomes the optimal option.
Now, let’s add a noise factor. You have back pain — how does that change our matrix?
From this, you can see that while before there were three good options and two sub-optimal ones, the addition of our noise factor means that the sub-optimal choices and one Good option are now off the table — leaving just one option if the weather is not looking fine and dry: take a coat!
Example: Three Trips to Barcelona
My disabled mother went to Barcelona three times. Most travellers grab a taxi from the airport without thinking twice. My mother decided to do the same as most people. Here’s how her trips went:
1st Time
- She went with an elderly friend.
- She got her handbag stolen at the airport taxi rank.
2nd Time
- Travelled on her own.
- She got in a taxi, and the driver threatened her until she paid more than the agreed fare.
3rd Time
- She travelled with me.
- We chose to get the 20-minute shuttle bus from the airport to the town centre, costing €4 each.
None of these experiences were positive in reality — but only the third one was not totally unpleasant, just a bit sub-optimal — which is often the choice faced by someone with physical disability or access needs.
The common taxi option didn’t work for her, and instead, we needed to find an alternative. While the shuttle bus was a less attractive option for many people, having the option gave my mother her optimal solution. If we hadn’t found an alternative mode of transport, she likely would have continued to have a poor experience. This is how users fall through cracks online when we only design a “default” option without alternatives.
How to use an inclusion-first mindset in your next experiment design
Follow these four steps to develop an inclusion-first mindset:
- Do a pre-mortem.
Identify all the potential sub-optimal outcomes of a test and the conditions that create them. - Create a matrix.
Use a matrix to get an overview of all the different combinations of conditions and the outcomes they lead to. - Add alternative options.
Actively create more solutions or add options to your matrix that lead to better outcomes for more users. - Look for interaction effects.
Systematically explore each permutation to uncover any potential interaction effects. For example, some users might take both an umbrella and a coat, causing unexpected consequences such as tripping over when holding both items.
To help you develop this inclusion mindset, think about how you can also apply these principles in daily life. For example, you can apply this thinking to any of the following scenarios:
- Event planning: Can I provide multiple options for sound, light, and environments?
- Team socials: Can I provide several locations, not just one?
- A day in the office: Can I create more seating options?
- Organising lunch: Can I provide 2+ cuisine and dietary options?
Conclusion
You may not have the “inclusion habit” yet, but it develops quickly with practice. If your ambition is to become a better experimenter, but you have not yet thought about inclusion, then you are missing out on a way to sharpen your craft and increase your value to the business, all while making your customers with access needs happier.
Download our Experiment Canvas to help set out your experiment design, or book time with me, Ellie Hughes, to discuss how to do this for your business.
