003 Everything You Need to Do Before You Can Run an A/B Test

Today’s episode is going to be about conversion rate optimization or CRO. Specifically, what do you need to do before starting an A/B test. We cover:

Steps you need to do before starting any test
KPIs to consider
Tools you can use
Why you should be thinking about CRO even before launching a site

Links in this episode:

Episode Notes:

There are no failed tests – only winners or learning opportunities. – VWO

When Should You Start Thinking About CRO?

It is never a bad idea to start thinking about CRO. However, it is much earlier than most people would say that you should start the CRO process.

If your website is not even live, and you’re about to launch for the first time, you know what key pages are, and key features in those pages. Prepare a measurement plan and a testing strategy around those features. The only thing you can be certain of is that you will not get it perfect on the first try.

And if you’re redesigning, all of that, plus use your existing website to get precious customer data, then use that to inform the decisions you make during the redesign and make your life easier.

How To Get the Best Results From Your A/B Test

Before I get into the numbers, testing is only one method used in CRO, among many, and definitely not the first one you should do on a project. Without proper research, how can you know what to test? You need to know where your audience struggles and the only way to learn that is through research.

You do the research, look at web analytics, maybe some heat maps, customer data, you do heuristic evaluation of the website, and you have a long list of issues you want to try to fix. Still not a good time to run tests. You need to know what to work on first, and for that, you need a prioritization framework. You want to know which issues you should start working on first.

Now you know what to work on first, and MAYBE it’s time to run an experiment. I mean, if it’s dead clear that something is broken, you will probably not test broken vs fixed, right? Or maybe you need to work on a hypothesis or do further research before you can run an experiment.

So it’s research, prioritization, then maybe testing. A CRO program does not start with “let’s test this”.

If you want to talk about actual numbers, it will depend on a lot of things, but if you don’t have let’s say 10 000 visitors to the page you’re testing and 500 transactions, per month, it’s probably better to focus your efforts elsewhere.

CRO Success Metrics

The ones that matter. On the business side of things, I don’t think CRO practitioners or agencies are the ones who should be setting the business metrics. But if you want to judge the CRO program itself, you can look at things like test success rate, testing velocity and revenue impact.

But if you look at long term value you get out of a CRO program, there’s even more, thanks to everything you learn about your audience and how you interact with your website. Even an experiment that fails can teach you what you need to do in the future and, equally important, what you need to avoid.

Which Pages Should You Start With?

No matter the website, it’s best to look at the entire conversion funnel. Where do the users land, what is their first experience with your website? What do you need them to do next? And what after that? Is there any friction along the way, anything that would make them nervous, you need to address all those issues throughout their journey.

Sometimes it will be simple, the user lands on your product page and buys the product straight away, but far more often that is not the case. Identify what your funnel, or funnels are, then start by identifying their weak points and fixing them.

HTML Skills Can Be Helpful When Running an A/B Test

I think everyone working with websites should have at the very least a basic understanding of how websites work, even if you’re not the person doing the implementation. But if you’re doing something like implementing A/B tests, you’re changing HTML of the page, in most cases CSS and even JavaScript, too.

Sure, there are platforms that let you use a visual editor, so you can make some changes yourself. But what happens when the visual editor is not good enough for the experiment variation you want to set up? This is why I think test implementation is best done by those who build the websites. There are people who can do both – plan the experiment and then implement it, but if you had to choose between a CRO specialist who can’t write any code or a developer who doesn’t understand CRO, or marketing, it’s probably better that the developer does the implementation.

How Long Should A/B Tests Run?

This is one of those things where experience doesn’t matter. It will depend on traffic volume, conversion rate, and the bump in conversion rate you get from the variation you’re testing against the original. The easiest way to learn this is if you use an A/B test statistical significance calculator, there are many of those available online.

Ideally, you want to run your experiment for at least one complete business cycle. In most cases, that will be one week. You need to do this because your traffic will be different from Thursday to Sunday than it is Monday to Wednesday. Not just the volume, but the quality of that traffic. You don’t want to make big decisions based on misleading test results. Even if your testing tool says you have statistically significant results before the week is up, you should wait.

Similarly, you can’t let your test run for too long. There was this joke Louis CK had years ago, he’d talk about a group of settlers migrating from the East Coast to the West Coast and they’d move at such a slow pace, of course, that when they arrive it would be a completely different group of people.

If the statistical significance calculator tells you you’d need to run an experiment for 4 months until it’s statistically significant, you can’t really run that experiment. A lot of external factors will change over that period. Seasons will change, your users will change and you will not be able to isolate those changes and know which one contributed to the difference in conversion rate. Ideally, you want to isolate the changes you’re making and limit the effect of any other changes as much as possible.

Just going back to my example earlier. I was working for a SaaS company and I’d say we had more than 100k+ visits per month. The problem we were trying to solve was how to increase engagements for our templates page. The end goal was to make it easier for our customers to use our app without starting from scratch.

We had a high traffic/ high performing page that had secondary navs on the left sidebar. So we decided to test and apply that format to our templates page hopefully to replicate the engagement.

So we set up the test on Optimizely, created another version of the page with the navs to the left sidebar instead of the top. Results were inconclusive even after a few weeks of running it. We ended up stopping the test and deciding to roll with version 2 anyway.

What could we have done better in this sample CRO experiment?

I think the biggest issue with that was that the change was not that significant. It was purely functional, so you were not trying to work on users’ motivation, but whether the user will be able to find the navigation in the page. And I’m not saying tests like that are bad, but you’re not really learning much about why one version works better than the other.

That’s what most people get wrong about experimentation, the first thing they’ll associate with A/B testing is button color. Even if you do get a clear winner, how does that help? If a blue button wins, does that mean making everything blue would be even better?

Try to learn something from every experiment you run. What is it that motivates your users to convert, what is it that holds them back? That’s the only way to continuously improve your website.

A/B Tests as a Process

CRO is an ongoing, systematic process, period. Its name may lead you to believe it is only about making conversion rate go up, but if your conversion rate goes from 5% to 15% because you just start offering 99% off on all products, how does that help you?

So, don’t just focus on making one number go up once, use CRO as a continuous process that helps you get to know your users, address their concerns, and ultimately deliver a better experience for them.

A CRO Success Story

I have a fun example. I was working with a client, an e-commerce website, and looking at their analytics I saw there were tons of users who would browse 5 or more, even 7 or more products in a session without adding anything to cart. So we thought, maybe they don’t know what to buy, they’re just experiencing paralysis by analysis. We created a top 10 product page, a simple landing page that lists best-selling products from the previous month, linked to it from the main menu.

This client had a fairly high conversion rate already, but when we added this top 10 page, users that visited it converted at a rate that was 3x compared to the site average. It easily became page with the highest page value, but what’s even more important was that it helped the users solve a problem they had. And that’s the best way to use CRO, as something that helps you help your audience.

What can you say about this: “Why fix something that isn’t broken!”

Because everything IS broken. Maybe not completely broken, but everything can always be improved. If you love your status quo, maybe your competitors are doing something to get better. Trends change, new technologies emerge, the only way to stay successful is by constantly evolving, and this is where conversion rate optimization can really help.

Recap and Final Thoughts

So there you have it, before jumping into an A/B test, remember to:

Do the research first (check out your tools – analytics, heatmaps and even qualitative data)
Set your business objectives – is it tied to revenue, or to address drop offs on important pages be clear on that
Prioritize – review your list and weigh in which is more important or which will bring you closer to your goals
Then consider an A/B test and build it to your process.

The title of this episode is “everything you need to do BEFORE you can run an A/B test”. If you’re just getting started with optimization of your website, trying to make it better for your users, or customers, A/B testing is not the first thing you should do, and I hope we were to convince you to focus on research and prioritization first. Testing is a way to fix a problem, but it’s not something that will show you what the problem is. Do those other things first, worry about testing later.

Don’t forget to subscribe to the No Hacks Marketing Podcast. And if you learned something new today, we would appreciate it if you can leave us a review on your favorite podcast platform.