When do we need user acceptance testing? And when can we get away without it?

User acceptance testing (UAT) is when your software goes in front of the user to get final sign-off — and when they ask for changes if not. In theory you shouldn’t need UAT at all (didn’t they tell you what they wanted? Weren’t you listening?), and indeed perhaps you can sometimes get away without it.

This argument is stronger if you’re considering UAT throughout the project, not just at the end. This would be the case if you’re UATing individual features, or individual deliveries in a culture of more and smaller iterations. The arguments are: more deliverables mean more UAT means more expense for the customer; smaller iterations mean a reduced chance of getting it wrong; frequent deliverables give the customer more opportunities to change anything they don’t like anyway.

However, even with small, frequent deliveries there are circumstances when it’s more important to do UAT. Looking back on some projects I’ve worked on, here are times when it was, or would have been, a good idea…

1. When the output of the software is subjective

I once worked on a project that asked what bands you liked and then based on that recommended albums by unsigned bands. This was a big leap away from the “people who liked that also liked this…” kind of recommendations, because the unsigned bands won’t have had a significant userbase, so we couldn’t rely on a self-generating wisdom-of-crowds mechanism. Plus, of course, music recommendation is highly subjective, and it was our clients who were the music experts — we implementers were mere software developers.

In this case you can see it was key for our clients to play with the system and be sure they were comfortable with it. If they didn’t then it would have meant tweaking the algorithm, not a complete software rewrite of the software. In the event something rather unexpected happened. While our client’s representative was very happy with the output he also found it rather disconcerting. He felt somewhat disenfranchised from the system because it seemed so mysterious. He realised that he couldn’t demonstrate it in front of his colleagues and their investors without having an answer for the suddenly inevitable question: how does it work? And once this was explained to hiim he became much more comfortable again.

If the output of the system had been much more objective and predictable this instance of UAT wouldn’t have been so important. In the end it threw up an important new acceptance criteria for client which we were quickly able to address.

2. When the software has a costly userbase

One of my past projects involved creating a user interface for a property data entry team. Individuals on the team were, as you’d expect, low-paid and largely interchangeable — after an hour of training you would know everything there was to know about the system. Your sole function was then to spend hour after unsociable hour entering information about properties you could never afford to live in for people you would never want to live next to.

But although the daily cost of individuals on the team was considered low, the cost of the team as a whole — and the value they were bringing to the business — was very high.

Problems with the user interface in this case would have had a knock-on effect across the whole team of 20 or 30 individuals, and cost their employer dear. It was key for someone to check not only that the system was responsive, but also that there were intuitive keystrokes across the data fields, data entry was reasonably forgiving, tabbing between fields happened in the order that data was presented in, and so on. In this case, the time and early feedback from one user at the vanguard saved time and cost 20 or 30 times over.

3. When you don’t have a QA department

A pretty obvious one, but if you don’t have people dedicated to quality assurance then those internal people who take on that function (probably the lucky developers or irritated account manager) will be too close to the software and too distant from client to pick up on all the issues that are important to them.

A good percentage of the projects I’ve worked on have involved integrating search engines. And in almost every instance when testing has been left to the development team they’ve alighted on one or two key phrases to use as test cases and considered that sufficient. By contrast, the end user to whom this is important will have half a dozen phrases that are relevant each day — and they’ll involve accented characters, non-ASCII characters, mismatched quote marks, and so on. And that’s before we get to the results page. Without effort the expert user has given the search engine a good workout — often raising difficult issues for the developers.

Actually, even if you do have a QA department then there’s a case for UAT, because more often than not the QA team will test against written criteria. There are very often criteria which, with the best effort in the world, just don’t get realised or written down at the requirements stage. And in these cases it’s only the end user who can really tell if it’s right.

4. When you lack detailed requirements

Another fairly obvious situation, but if you’ve not had a good bit of customer input at the start of the project, then UAT is going to be even more important later on.

A sales person I worked with once had a cunning plan. We’d already produced an e-commerce site for one of our clients, and he decided to sell a duplicate system to one of their rivals. It was good plan on paper. There was no contractual restriction for us; the new client would get their site at a relatively low cost; they’d get it quicker than usual; and we’d only have to reskin it. The deal was done over a nice lunch.

You can guess what happened in reality. When they saw what we imagined was a near-complete version they wanted some changes. The checkout system wasn’t quite to their liking; their product catalogue didn’t have quite the fields we were expecting; the order placement system didn’t align with what they had internally; and so on. It cost everyone more than they had anticipated and it was delivered later than anyone would have liked.

Certainly this is a strong case for robust specifications — or at least understanding what you’re getting into before you get into it. But it also demonstrates that lack of early information cost us dearly when the information did come along later. We hadn’t made time for UAT, which is why we delivered the final product later than planned. Fortunately for everyone there was no externally-driven deadline. If there had been then we’d have been obliged to deliver something which the client wasn’t happy with.

5. When the inputs to the software are difficult to specify

Finally an example that brings us right to the present.

The current work to refresh the Guardian Unlimited website is much more than applying a lick of paint. We’re also producing new tools for our reporters and new ways they can tell their stories. But telling a news story is not simply a matter of filling in a few boxes and clicking Save. To make a story relevant requires a lot of hard training. To edit an entire news section additionally needs an awful lot of experience and good intuition…

As I write, MediaGuardian.co.uk is leading with three big stories and a podcast, plus some smaller items headed “More media news”. The stories have between three and six sublinks which indicates the depth of related news that the editor has available. Yet at the same time the Technology section is showcasing six stories, all with a bit of blurb, but only one of which has sublinks (to a video and a gallery). In both cases the editors are using the same system, but the page layout has to balance itself out regardless of the highly variable input.

But though the input is highly variable it is not without bounds. A section would never be showing just one story, and an editor would almost certainly not try to highlight twenty stories and claim they had largely equal weight. The only way to really know the bounds in which the system is supposed to reasonably work is to sit an editor down with the tools and let them lay out pages in response to a real day’s stories. No amount of requirements analysis and QA experience can substitute for a real journalist responding to a real day’s events.

And that’s the beginning…

No mechanism should be applied blindly; there are times when user acceptance testing is more appropriate, and times when it is less appropriate. The important thing is to be aware first that UAT exists, and second that there are criteria which you can apply to assess how valuable it can be.