Risk and uncertainty, Working practices

Prioritising hypothesis tests

A little while ago I was working with a team that had a lot of user research to do. Naturally, we only had limited time, so we needed to decide how to prioritise our work. This is how we did it.

First of all we wrote out all the ideas we wanted to examine. Some of these were actual hypotheses—do our users respond better to this or that? If we did X would they conclude Y? Some of the things we wanted to examine were more general—what do our users think of this way of presenting information? If we offered them five alternative ways of doing something, what would we discover were the strengths and weaknesses of each. And so on.

Next we drew two axes and positioned each idea somewhere in the x/y space. The x-axis ran from 0% to 100% and represented our confidence in the result. If we were certain our users would hate it we scored it at 0%. If we were certain they’d love it we scored it at 100%. Something at 75% meant we were 75% sure people would like it, but had about 25% reservation. Anything we were wholly unsure about was 50%.

The y-axis ran from 0 to an arbitrary 10. It represented impact on our goals, on some made up and unspecified scale. If we thought an idea would have no impact we’d put it at 0 on the y-axis. If something had middling impact it would score 5. A score of 10, of course, was maximum impact.

Once we had everything positioned we could prioritise the testing of our ideas much more effectively. The highest priority tests were for anything closest to 50% on the x-axis and maximum on the y-axis. These were things where we had no sense of how people would respond, but knew getting it right would have the biggest impact. The further anything was from the (50%, 10) point the lower its priority for testing.

Of course, this is all based on estimation and an arbitrary impact measure, but it gave us a good step forward for prioritising our user research tests.

Advertisements

Discussion

2 thoughts on “Prioritising hypothesis tests

  1. Sounds like Impact Estimation.

    Posted by Niels | 13 March 2019, 8:54 am
  2. That’s interesting – that’s not something that occurred to me. It definitely has “impact” and “estimate” in its description. But I see some key differences: there are two defined “requirements” (confidence in the outcome of a test, and impact), rather than an arbitrary number; we are seeking to prioritise all our options, rather than selecting the best one or few; this is very much geared towards testing hypotheses, rather than selecting solutions (which is what impact estimation is mostly-but-not-exclusively used for). Maybe this is a very particular form of impact estimation.

    Although I would say if this is like impact estimation then lots of other things are more like impact estimation. It would be interesting to collect those together. if one was so inclined.

    Posted by Nik | 13 March 2019, 9:11 am