How to run good experiments

In a previous post, we explored how to test messages and expressions of your value proposition with the help of Will Chan from Scale Messaging. Will gave us a practical approach to taking these messages and testing them in order to drive improvements in conversion.

In this post, we'll outline two important qualities of any business going down the path of experiments to validate its business model: Discipline & Creativity

If you're familiar with our work, you'll know that we are big fans of the lean startup approach who encourage us to focus on traction metrics, not vanity metrics. Conversion rates tend to be a great example of a traction metric as they are based on the relation between two variables.

There are a bunch of ways you could measure conversion. Here are some examples:

• Conversion on Physical Channels:

  • Outbound meetings requested to meetings completed
  • First meetings to proposal written
  • Referrals from past customers to new prospects

• Conversion on Digital Channels:

  • Visitors taking an action on your site
  • Audiences clicking on the CTA of your ad
  • Registrations to an event or newsletter

If you're not sure about how you can measure conversion, Pirate Metrics (AARRR) are a good starting point:

Source: Product Frameworks

Granted - testing on digital channels is fast and easy given advertising platforms like Facebook, but testing any of these conversions is doable if you approach it with some discipline and creativity.

Discipline:

Strategyzer's test card provides a useful framework to think through an experiment and its objectives before launching into it

1. Hypothesis

   • What is the belief behind the test? What do you believe to be true? This is just a starting point for further investigation. For example:
   • We believe that Climate Tech founders want to learn from experts and from case studies that can help them validate a product before building it.

2. Test

   • What is the experiment you will conduct to test the hypothesis? How much will it cost? How reliable is the data? For example:
   • To verify that, we will create a landing page where people can register for a webinar on the topic

3. Metric

   • What is the measure you will collect from the experiment? How long will you need to collect the data? For example:
   • And measure how many of the visitors to the landing page convert into webinar registrations over the course of a week.

4. Success Criteria:

   • How will you know if your hypothesis is right? What are the minimum success criteria? For example:
   • We are right if at least 30% of visitors to the landing page convert into registration

Creativity:

You need to believe that there is a way in which you can test your hypothesis. Take some time to think through the channel and approach that makes the most sense to your customer's journey. It's less relevant to test conversion on a landing page if most of your buyers are coming from foot traffic. Be creative here.

Building on the examples above, here are some ideas on how you could test conversion:

• Conversion on Physical Channels:

  • Outbound meetings requested to meetings completed
    • Send emails, inMails or any other one-to-one message with different key messages and measure how many respond.
  • First meetings completed to proposals presented
    • Use different versions of pitch decks with different key messages and value propositions targeting different customer segments and measure close rates
  • Referrals from past customers to new prospects
    • Offer different referral deals (X% off your next purchase, etc) to your existing customers to encourage referrals

• Conversion on Digital Channels:

  • Visitors taking an action on your site
    • Add a new button, registration, product, etc to your site and measure whether they take the new action. You can pair this with a new source to drive incremental traffic
  • Audiences clicking on the CTA of your ad
    • Run ads with different key messages, different creatives or different calls to action.
  • Registrations to an event or newsletter
    • Create landing pages with different topics and measure which one has the best conversion

This is where a bit of hustle can go a long way. It's easy to hide behind experiments on digital channels, but a conversion metric will give you no clue about the reason behind the user's behaviour. WHY didn't the experiment achieve the success criteria?

I always combine some data-rich quantitative tests with some feedback-heavy qualitative ones. What's more, it'll be really hard to have a decent starting hypothesis unless you've had qualitative customer conversations to give you insights to start your testing. So make sure you're including customer conversations in your approach.

David Bland is an authority on designing experiments and has a great book on the topic. Here is an extract from the book with even more experiment examples:

Source: Strategyzer