Creating a good Website Testing Plan

This series on online testing was originally published in 2013 on my old blog, The Big Picture, and updated for Hickam’s Dictum. It should also be noted that Oracle purchased Maxymiser in 2015, two years after this article was first published.

In part 1 of this series we established what the different testing options are (A/B vs. MVT) when it comes to digital content, with a focus on website testing. Now lets talk about process. It’s very easy for your testing program to become a nightmare down the road if you do not have the proper structure in place or ask the right questions. Process strategy is very important so you make the most efficient use of your time and money.

Establishing the Hypotheses

The number one thing to establish before embarking on a testing program, is what question (or questions), you are trying to answer. Select the right thought leadership to define your questions and hypotheses (typically these should include analysts, marketing strategy leads and developers). You should establish your questions and hypotheses before running any tests and/or onboarding a vendor/testing partner. Your questions should also have educated guesses/answers (hypotheses) to be tested. Example:

Question — Will changing the color of the purchase button on a product page increase sales?

Answer (Hypothesis) — Yes

Your hypotheses should fall into categories geared at increasing the value proposition of your website. So categories like increasing revenue, improving engagement, moving traffic through the funnel, even assessing how a change on the website will affect internal (website) or external (SEM) searches should be considered. This is by far the most important step in the process, as it will determine the test scope and budget. Every stakeholder should be consulted for a list of hypotheses to be tested, and the total questions and hypotheses should be written down as part of the testing process documentation. The reason for this is (A) to have legacy documentation on hand, but more importantly (B) to guide the conversations that will eventually happen between your team and the selected vendor. I strongly recommend that you have wire-frames on hand as you create your hypotheses, so you can visually assess how your testing might affect the user experience. This part of the process will also determine the types of tests you will invest in. So go through your website with any relevant stakeholders (e.g. analysts, channel leads, etc.) and determine what you want to test! And remember to include different device types (desktop, mobile, tablets, TV) and other segmentation criteria (e.g. demographics, browser type) in your hypotheses formation so you understand how user experiences differ across mediums!

Building the Testing Plan: Assigning Categories

Now that you have established the hypotheses to be tested and you have them in writing, its time to scope out your test. Testing should never be thought of as one time events, but rather as an ongoing culture. With that being said, take some time and review all the hypotheses you and your team created. Then split your list of hypotheses into two categories: one for A/B testing, and one for MVT testing. If you need a refresher on the differences between A/B and MVT testing, read the first post in this series. Breaking the tests out by category as you plan will come in handy when aligning on costs with any testing vendor. Also put this step in writing (I recommend using excel).

Building the Testing Plan: Estimating Traffic and Building a Testing Calendar

You’ve made a list of hypotheses to test, and split them into categories. Now its time to plan. If you are a business, you likely are running multiple campaigns every quarter, that are attracting different audiences and levels of traffic to your website via different channels. Having a good idea of the levels of traffic that will swing by your site by campaign and during BAU periods is important because it will help you understand the best periods to use different tests. Remember, MVT testing requires more traffic than A/B testing as with MVT testing we are assessing multiple elements, which require a larger sample size. This means we will need a longer time to build up that sample. A/B testing by contrast, requires less time to draw conclusions. Estimate your traffic levels. Most website testing tools will let you know if your sample is statistically significant, and also offer sample size calculators that you can use for planning (here is one for Optimizely). Estimate your traffic, and build out your testing calendar keeping in mind the points I just mentioned. Having such a document in place will enable you and your team to have productive and efficient on-boarding sessions with any testing vendor. To illustrate, the below image is a very basic testing calendar I created for a car manufacturer website:

Click to enlarge

Building the Testing Plan: Selecting the Right Testing Solution

You’ve established your hypotheses for testing, assigned them to categories by test type, estimated your site traffic and built a testing calendar. Now you are armed with enough information to properly vet the testing solutions on the market. Since you’ve done your homework by creating your hypotheses and testing calendar, each vendor considered should be able to give you an accurate estimate and insights on items like:

• costs,
• reporting output,
• data quality/accuracy
• traffic allocation options,
• the user friendliness of their platform,
• their customer service options,
• integration capabilities,
• personalization options,
• their device testing capabilities (including mobile and TV apps),
• configuration options,
• how testing will affect your website performance,
• if they use sequential testing,
• real time alerts
• their reporting/analytics capabilities,
• user spoofing
• server side and client side optimization
• how the solution works with firewalls
• and so on.

Having a solid website testing plan in place will enable you and your team to make more efficient use of your time and money, and help mitigate future risk. The key is to have the hypotheses, draft of testing calendar and thought leadership in place before onboarding a testing partner.

For the next post in this series we’ll profile some of the top website testing solutions in the market.

I hope you enjoyed this article! Please click the applause button 👏below or on the side so others can learn about this article as well! And please Bookmark and/or Follow Hickam’s Dictum for more actionable strategic insights!