Your Test is Only as Good as Your Hypothesis How to Cra! Solid, Data-Driven Hypotheses
ConversionXL Live 2015
@ContentVerve
If we get a responsive website, we’ll make more money…
@ContentVerve
@ContentVerve
$$$ Cost of responsive redesign $$$ $$$ Serious loss of revenue $$$ $$$ Cost of “damage control” $$$ $$$ Cost of double work $$$
@ContentVerve
@ContentVerve
I’m a split test junkie!!!
@ContentVerve
Dumbass…
@ContentVerve
A solid test hypothesis is an informed solu"on to a real problem - not an arbitrary guess.
@ContentVerve
It’s the star"ng point for further research.
@ContentVerve
1. The change you want to make 2. How this change will affect your prospects 3. The impact you expect to see as a result
@ContentVerve
1. The change you want to make 2. How this change will affect your prospects 3. The impact you expect to see as a result
@ContentVerve
X into, ______ X I can get more By changing ______ X and thus increase ______. X prospects to ______
@ContentVerve
Hardwire Your Brain to Think in Hypotheses
@ContentVerve
Think in terms of “op"miza"on hypotheses” not “test hypotheses”
@ContentVerve
And now….. A Really Stupid A/B Test!
@ContentVerve
@ContentVerve
@ContentVerve
@ContentVerve
Control:
Treatment:
2 months data & 490 downloads
@ContentVerve
Control:
Treatment:
2 months data & 490 downloads No Real Difference
@ContentVerve
@ContentVerve
@ContentVerve
40% Check prices
37% Learn more
23% Book
@ContentVerve
40% Tjek prices
37% Learn more
23% Book
@ContentVerve
By adding a pricing page, I can bridge a serious gap in the decision-making process, mo"vate more prospects to say yes and thus increase the number of leads.
@ContentVerve
@ContentVerve
X into, ______ X I can get more By changing ______ X and thus increase ______. X prospects to ______
@ContentVerve
Total number of days to run the test: 314 days
@ContentVerve
If I can get a li# of 33%
Total number of days to run the test: 35 days
@ContentVerve
@ContentVerve
@ContentVerve
@ContentVerve
By adding a privacy policy, I can reduce fric"on, get more prospects to give me their info, and thus increase sign-ups.
@ContentVerve
Control:
Treatment:
100% privacy - we will never spam you!
@ContentVerve
Control:
Treatment:
18.70% drop in sign-‐ups Sta$s$cal Confidence Level 96%
@ContentVerve
By removing the word “spam” and adding a guarantee, I can reduce fric"on, get more prospects to give me their info, and thus increase sign-ups.
@ContentVerve
Control:
Treatment:
We guarantee 100% privacy. Your information will not be shared
@ContentVerve
Control:
Treatment:
19.47% increase in sign-ups Sta$s$cal Confidence Level 96%
@ContentVerve
By removing the guarantee, and adding the word “spam” I can increase fric"on, get less prospects to give me their informa"on, and thus decrease sign-ups.
@ContentVerve
@ContentVerve
24% drop in downloads Sta$s$cal Confidence Level 99%
1 month of data / 349 conversions @ContentVerve
1. Conduct Research
2. Hypothesize
3. Create Treatment
4. Test Treatment
5. Analyze
6. Run Follow-Up Experiment @ContentVerve
1. Conduct Research
2. Hypothesize
3. Create Treatment
4. Test Treatment
5. Analyze
6. Run Follow-Up Experiment
@ContentVerve
1. Conduct Research
2. Hypothesize
3. Create Treatment
4. Test Treatment
5. Analyze
6. Run Follow-Up Experiment
@ContentVerve
Conversion Research
@ContentVerve
Device Category
Image Removed
@ContentVerve
Browser
Image Removed
@ContentVerve
Browser - Desktop Sessions
Image Removed
@ContentVerve
Other reports: Top landing pages Site Search New vs. returning Nav overview Site Speed Unbounced sessions Unbounced pr. device New vs. returning pr. device Learn to use advanced segments!
@ContentVerve
Slides Removed
Home Page
Category Page
Product Page
Cart
Checkout
Confirma"on
@ContentVerve
Home Page
Category Page
Product Page
Cart
Checkout
Confirma"on
@ContentVerve
By reorganizing the product page content, I can create a more logical user expereince, reduce cogni"ve fric"on, get more prospects to configure a product and thus increase purchases.
@ContentVerve
@ContentVerve
Top-Selling Product Categories
@ContentVerve
Top-Selling Products
@ContentVerve
@ContentVerve
Slide Removed
1.
2.
80% drop off
3.
54% drop off
4.
14% drop off
5.
66% drop off
6.
4% drop off
@ContentVerve
How to Build Funnels Manually Go to “All Pages” in GA
Type URL Structure
Write Down Unique Pageviews
@ContentVerve
@ContentVerve
@ContentVerve
@ContentVerve
“In the economy of ac"on, effort is a cost… Laziness is built deep into our system.” - Daniel Kahneman
@ContentVerve
@ContentVerve @ContentVerve
@ContentVerve @ContentVerve
@ContentVerve @ContentVerve
Al?
@ContentVerve
Michael
[email protected]
Al? Al? Daddy?
@ContentVerve
@ContentVerve
@ContentVerve @ContentVerve
Run Clean Tests, So You Are Acutally Ge$ng Insight!
@ContentVerve
@ContentVerve
@ContentVerve
Slide Removed
Qualita"ve Research
@ContentVerve
Interviews with: - Customers / Poten"al customers - Sales - Customer service / support
@ContentVerve
Interviews with:
- Customers / Poten"al customers - Sales - Customer service / support
@ContentVerve
Sales 1. What are the top 3 ques"ons you get from poten"al customers? 2. What do you answer to these ques"ons? 3. What is the biggest barrier keeping people from buying your product? 4. Are there any specfic selling points that work par"cularly well? If so, which ones? 5. Have we missed anything important? Got something to add? @ContentVerve
Customer service / Support 1. What are the top 3 ques"ons you get from poten"al customers? 2. What do you answer to these ques"ons? 3. Are there any par"cular aspects of X that people don’t understand? 4. What aspects of X do people like the most/least? 5. Have we missed anything important? Got something to add? @ContentVerve
Does A/B Tes"ng Make Sense on Your Website?
@ContentVerve
Your Test is Only as Good as Your Hypothesis The be%er the research, the be%er the insight The be%er the insight, the be%er the hypothesis The be%er the hypothesis, the be%er the test The be%er the test, the be%er the learning
@ContentVerve
Don’t be the Split Test Junkie
Be The Conversion Plumber
@ContentVerve
Thank You For Listening! ConversionXL Live 2015
