If you’re developing a digital product in a smart way you’ll want to be tracking the metrics from the features or changes you introduce. But how do you know what a “good” result looks like?
For example, you may want to test your assumption that people will sign up for your service. You may even do some A/B testing to find out if a blue Sign-Up button is more or less effective than a red one. But how do you determine if the results are a Yes? After all, if you put it in front of enough people, some of them are bound to sign up. And if 5% of blue-button users sign up compared to 7% of red-button users, does that mean red is better than blue, or that too few people are signing up either way? What you absolutely cannot do is decide the success threshold after the event—you have to decide what success looks like before you start the experiment. But how?
One way to answer this is to ask what the business plan requires. You can’t expect to achieve your business plan’s goals in one experiment, but the plan should show some kind of growth curve, and the question becomes whether you can match that. (Most people talk about “hockey stick” growth, but I think that depends on the metric. A hockey stick may be important for, say, raw number of users, but I’m sure a linear growth for average revenue per user would be a good plan for many businesses.) If your plan requires 6% growth month on month, and you’re tracking 3-4% then it’s time to do some serious rethinking. Equally, if you’re ahead of the plan, but the plan requires exponential growth while your actual growth is linear, then again we can’t pat ourselves on the back.
Another way to answer the question is to ask what the figure is for comparative products. What is the sign-up rate for a direct competitor—or even a distantly-related product? This will at least tell you whether you should consider, say, 2% or 20% to be a good figure. If a similar product is getting a 10% sign-up rate then before you go into the test you should acknowledge that you should not consider 5% to be a good outcome.
Determining “good” needs to be done before the numbers come in. And there are useful guides out there if you look in the right places.
niksilver.com 