When do we need user acceptance testing? And when can we get away without it?

User acceptance testing (UAT) is when your software goes in front of the user to get final sign-off — and when they ask for changes if not. In theory you shouldn’t need UAT at all (didn’t they tell you what they wanted? Weren’t you listening?), and indeed perhaps you can sometimes get away without it.

This argument is stronger if you’re considering UAT throughout the project, not just at the end. This would be the case if you’re UATing individual features, or individual deliveries in a culture of more and smaller iterations. The arguments are: more deliverables mean more UAT means more expense for the customer; smaller iterations mean a reduced chance of getting it wrong; frequent deliverables give the customer more opportunities to change anything they don’t like anyway.

However, even with small, frequent deliveries there are circumstances when it’s more important to do UAT. Looking back on some projects I’ve worked on, here are times when it was, or would have been, a good idea…

1. When the output of the software is subjective

I once worked on a project that asked what bands you liked and then based on that recommended albums by unsigned bands. This was a big leap away from the “people who liked that also liked this…” kind of recommendations, because the unsigned bands won’t have had a significant userbase, so we couldn’t rely on a self-generating wisdom-of-crowds mechanism. Plus, of course, music recommendation is highly subjective, and it was our clients who were the music experts — we implementers were mere software developers.

In this case you can see it was key for our clients to play with the system and be sure they were comfortable with it. If they didn’t then it would have meant tweaking the algorithm, not a complete software rewrite of the software. In the event something rather unexpected happened. While our client’s representative was very happy with the output he also found it rather disconcerting. He felt somewhat disenfranchised from the system because it seemed so mysterious. He realised that he couldn’t demonstrate it in front of his colleagues and their investors without having an answer for the suddenly inevitable question: how does it work? And once this was explained to hiim he became much more comfortable again.

If the output of the system had been much more objective and predictable this instance of UAT wouldn’t have been so important. In the end it threw up an important new acceptance criteria for client which we were quickly able to address.

2. When the software has a costly userbase

One of my past projects involved creating a user interface for a property data entry team. Individuals on the team were, as you’d expect, low-paid and largely interchangeable — after an hour of training you would know everything there was to know about the system. Your sole function was then to spend hour after unsociable hour entering information about properties you could never afford to live in for people you would never want to live next to.

But although the daily cost of individuals on the team was considered low, the cost of the team as a whole — and the value they were bringing to the business — was very high.

Problems with the user interface in this case would have had a knock-on effect across the whole team of 20 or 30 individuals, and cost their employer dear. It was key for someone to check not only that the system was responsive, but also that there were intuitive keystrokes across the data fields, data entry was reasonably forgiving, tabbing between fields happened in the order that data was presented in, and so on. In this case, the time and early feedback from one user at the vanguard saved time and cost 20 or 30 times over.

3. When you don’t have a QA department

A pretty obvious one, but if you don’t have people dedicated to quality assurance then those internal people who take on that function (probably the lucky developers or irritated account manager) will be too close to the software and too distant from client to pick up on all the issues that are important to them.

A good percentage of the projects I’ve worked on have involved integrating search engines. And in almost every instance when testing has been left to the development team they’ve alighted on one or two key phrases to use as test cases and considered that sufficient. By contrast, the end user to whom this is important will have half a dozen phrases that are relevant each day — and they’ll involve accented characters, non-ASCII characters, mismatched quote marks, and so on. And that’s before we get to the results page. Without effort the expert user has given the search engine a good workout — often raising difficult issues for the developers.

Actually, even if you do have a QA department then there’s a case for UAT, because more often than not the QA team will test against written criteria. There are very often criteria which, with the best effort in the world, just don’t get realised or written down at the requirements stage. And in these cases it’s only the end user who can really tell if it’s right.

4. When you lack detailed requirements

Another fairly obvious situation, but if you’ve not had a good bit of customer input at the start of the project, then UAT is going to be even more important later on.

A sales person I worked with once had a cunning plan. We’d already produced an e-commerce site for one of our clients, and he decided to sell a duplicate system to one of their rivals. It was good plan on paper. There was no contractual restriction for us; the new client would get their site at a relatively low cost; they’d get it quicker than usual; and we’d only have to reskin it. The deal was done over a nice lunch.

You can guess what happened in reality. When they saw what we imagined was a near-complete version they wanted some changes. The checkout system wasn’t quite to their liking; their product catalogue didn’t have quite the fields we were expecting; the order placement system didn’t align with what they had internally; and so on. It cost everyone more than they had anticipated and it was delivered later than anyone would have liked.

Certainly this is a strong case for robust specifications — or at least understanding what you’re getting into before you get into it. But it also demonstrates that lack of early information cost us dearly when the information did come along later. We hadn’t made time for UAT, which is why we delivered the final product later than planned. Fortunately for everyone there was no externally-driven deadline. If there had been then we’d have been obliged to deliver something which the client wasn’t happy with.

5. When the inputs to the software are difficult to specify

Finally an example that brings us right to the present.

The current work to refresh the Guardian Unlimited website is much more than applying a lick of paint. We’re also producing new tools for our reporters and new ways they can tell their stories. But telling a news story is not simply a matter of filling in a few boxes and clicking Save. To make a story relevant requires a lot of hard training. To edit an entire news section additionally needs an awful lot of experience and good intuition…

As I write, MediaGuardian.co.uk is leading with three big stories and a podcast, plus some smaller items headed “More media news”. The stories have between three and six sublinks which indicates the depth of related news that the editor has available. Yet at the same time the Technology section is showcasing six stories, all with a bit of blurb, but only one of which has sublinks (to a video and a gallery). In both cases the editors are using the same system, but the page layout has to balance itself out regardless of the highly variable input.

But though the input is highly variable it is not without bounds. A section would never be showing just one story, and an editor would almost certainly not try to highlight twenty stories and claim they had largely equal weight. The only way to really know the bounds in which the system is supposed to reasonably work is to sit an editor down with the tools and let them lay out pages in response to a real day’s stories. No amount of requirements analysis and QA experience can substitute for a real journalist responding to a real day’s events.

And that’s the beginning…

No mechanism should be applied blindly; there are times when user acceptance testing is more appropriate, and times when it is less appropriate. The important thing is to be aware first that UAT exists, and second that there are criteria which you can apply to assess how valuable it can be.

There’s more to software development than getting a software project delivered to the users’ satisfaction, on time and on budget. (As if that wasn’t enough.) Sometimes someone has to ensure the system lasts for the long term. But when that happens it’s a balancing act: between the project and the product.

The project and the product might start at the same point, but they usually finish at very different points. A project finishes with the last milestone release of the code — and a party. The product finishes with the opposite: the last byte of code being removed from the servers — and a party only if it was really nasty software.

The balance between project and product is something that I see again and again in many different forms. Here are some examples from different stages of the project lifecycle…

Tools: Java verus Ruby

There’s an amusing little spat going on over at Michael Coté’s blog. It’s a fairly traditional tussle of Java versus Ruby (and other scripting tools), in which Coté accuses Java people of being overly concerned with internal architecture, while Ruby (and LAMP and Django) people are more concerned with delivering the goods. But Robert Cooper adds the telling observation that the maintenance of the delivered goods could well be a nightmare if it comes from the scripters, because it often involves a “mish mosh” of technologies with non-standard dependencies.

Robert’s point, more generally, is that getting the application out the door is only part of what a software team needs to worry about, and the choice of tools has a bearing on life beyond the project end.

People: Internal and external

Having an internal software team is a wonderful thing. Among other things you can build up a huge amount of expertise, both technical and business-specific, which helps everything go much more smoothly. But sometimes bringing in consultants from out the outside has a benefit, too, and beyond mere extra bodies. Consultants get to see many different projects, technologies, businesses and people over a very short time, and they bring this accumulated expertise with them.

Mixing internal and external people like this can work really well. In terms of ideas, external people can bring ideas and propose directions that might otherwise not have been considered. Internal people are all too aware of the long term implications of these ideas in their organisation and can stop the crazier ones before they get too far. In terms of making the best use of time, external people are often particularly driven by project deadlines, perhaps because they’re often only employed for projects, so it’s all they ever know. Internal people can see the time spent now but also have a very good feel for time likely to be spent in the future, so can determine better when additional time within a project is actually investment for the future.

Sometimes work is outsourced entirely to external teams. Sometimes work is undertaken entirely by internal people. When the two kinds of people mix there is creative conflict, and the balance between project and product is tested.

Release: Releases rarely happen once

I once worked on a six month project with a big bang release. At the end of the project was a single release which took three days. That wasn’t three days of integration, testing and launch. That was three days from the point when the software was entirely ready for the end users, to the point where it was actually available to those users.

In terms of the project this was fine: on the scale of a six month project, a three day release is negligible. But when we came to release 1.0.1 and release 1.0.2 it was a bit excessive, and by release 1.0.16 it was actually a serious business issue.

There are many lessons to be learnt from this, but one is that we had erred towards the project too much and our lack of consideration for the product caused us problems very, very quickly.

Soup to nuts

It’s too easy to forget about the product in the rush to the end of the project. At one time I realised I was so negligent in this area I stuck a large message on my monitor in 36 point font:

…and how will that be maintained?

Balancing project versus product affects everything from who you involve at the start through to how you hit your deadlines and on to how it affects the rest of your working life (and possibly your non-working life if you find yourself on call). There isn’t a one-size-fits-all answer, but at least if we are aware of the tension between the project and product we have a better chance of making more informed decisions.

Quality seems to be widely misunderstood. People often talk about quality being a key variable that a project team can choose to alter to reshape their project — along with other variables such as cost, scope and time. I think that view is flawed. Meanwhile Mishkin Berteig is not unusual when he says “I have come to believe that quality is not negotiable” — meaning that although it’s possible to negotiate over it, it’s fundamentally wrong to do so. I think that view is flawed, too. From my position quality can rarely vary in the same way as other key project variables, is rarely changed directly, and often it is right to allow it to drop below 100%. I want to spend some time here explaining why.

Before I launch into this, allow me to say that although this is inspired by an issue on which I differ with Mishkin, I do have great respect for him. In fact, it’s arguable that I’m only writing this because I have such respect for him — because it’s so surprising to see him write something that seems to be so mistaken.

The sections ahead are:

• Quality is about much more than bugs
• Quality is rarely addressed at the outset
• Quality is less flexible than scope, time or money
• Quality is usually the driven, not the driver
• Quality can be lowered — and sometimes that’s okay
• Non-executive summary for non-executives

Now, let’s get on with things…

Quality is about much more than bugs

First, let’s make one thing clear: quality is not just bugs. It includes look and feel, ease of use, ease of installation, performance, and so on. Looking at Mishkin’s post it’s clear to me that we he says “quality” he really means “bugs”: he is making a strong case for test-driven development. But test-driven development doesn’t help with an application that requires six clicks rather than one to enter the default data. It doesn’t help that one of your modules needs administrator access to save its state. It doesn’t help that you can’t add any more servers when you need to triple its capacity. These are situations where quality could be improved, but which aren’t solved by test-driven development. Quality is wider than that.

Quality is rarely addressed at the outset

There’s a classic project management mantra in many variations: “cost, quality, time: pick any two” is one such [1, 2 (PDF), 3]. At one level it implies (quite fairly) that the major project variables at the outset are interdependent, and you can’t choose settings for them all. You can’t say, for instance “I want it to do all this, I want it to be flawless, and I want it in a week” — making just two of those demands determines what you get for the third.

But at another level it suggests that people are making these kinds of decisions, and that quality is often determined at the outset. I find this implausible. I rarely hear clear requirements at the start of a project about the quality of the result. I’ve heard vague pronouncements, but rarely anything that could be factored into a project plan. I’ve heard “we need to do better than last time”, and “let’s not over-do it the way we did before”. But that’s usually part of an agenda-setting speech, not the project plan. It’s not enough to help make a calculable impact on scope, cost or time.

At the start of a project you will hear a precise number for a budget, a strong date for a deadline, and (we hope) a clear list of requirements. But it’s very rare that people give a precise level of quality — it doesn’t even have a common unit of measurement (remember: quality is not just a bug count). Quality is rarely negotiated at the outset. The client expects the software to do the job specified, and that’s that.

Quality is less flexible than scope, time or money

Another misconception is that quality can be varied to the same degree as other major (initial, or input) variables such as scope, time and money. Scope can be cut right back, or added to indefinitely. Similarly for money and time. But quality doesn’t vary in the same way. Quality can’t be increased arbitrarily — after a point there’s nothing left to do, assuming the scope isn’t changing. Similarly it can’t be reduced to zero, because below a certain point — well before you get to zero — the product isn’t doing the thing it was supposed to do and therefore you’re reducing scope.

However, while I believe that quality varies less than other project variables, I do believe the consequences of improving quality — even a small amount — are significant. Unit tests reduce firefighting and ensure the project is always moving forward. Usability testing ensures people are much more comfortable and confident with your software, paying dividends throughout the business. Increasing quality by a small amount can have a big, positive knock-on effect.

Quality is usually the driven, not the driver

I’ve talked about scope, time and money being “input” or “initial” variables. They’re things that are usually explicity set at the start of the project. Quality, however, might be set early on, but it’s usually determined by other things.

Sam Newman writes, correctly in my opinion, that quality is driven by process. Here, they way you do things determines how good the result is. Quality is an “output” variable more often than not. Introducing test-first development is a process change; allowing post-development testing time to be reduced because the development phase has overrun is a consequence of an unfortunate project management process; introducing or eliminating performance testing is a process change. In all these cases quality is not an independent variable; it is driven by other things.

Quality can be lowered — and sometimes that’s okay

Quality is negotiable, and it would be foolish if it wasn’t. It would be foolish if budget, scope and time weren’t ever negotiable, but they are, and in that respect quality stands next to them on a level.

Real situation: The website is being built; the investors are making bullish noises in the City; the launch — a release and public demo — was scheduled months ago and is due next week with 200 big media attendees, and testing is still on-going. If a real, but infrequent, bug is found that can’t be fixed by next week does the company cancel the launch?

Real situation: You’ve been working for months on your release. It’s almost done, but there’s another project looming, which the sales team are desperate get started. The time remaining for your current work will leave some untidy edges, but there’s a pressing commercial need to release it and move on to the new project before the company loses more competitive edge than it already has. Do you push for a project extension?

If a developer tells me their professional needs are bigger than those of the organisation in which they work, then they shouldn’t be working there. A developer works for their organisation — just as everyone in that organisation does — not the other way round. And making the right decisions for the business must have positive consequences for the development team.

Mishkin says

Every time the developers sacrifice quality for schedule/cost/features, they incur a debt. Sometimes this is called “technical debt”. I like to call it lying.

Powerful prose, but I’m afraid that particular extract strikes me as uncharacteristically arrogant. “Technical debt” is an honest term. By using this term the developer says to their organisation that the decision taken now will incur an increased cost later. The organisation as a whole can then weigh up the options and make an informed decision. And call me crazy, but it might just be that after being appraised of the situation, the project board, or managers, or directors might be in a better position to make a strategic decision than the developer who’s paid for their expertise in one very particular part of the organisation.

Of course, that’s not to say that every decision on quality needs to be raised to board level. Just as a developer wouldn’t dream of bothering their project board as to whether they should use descriptive variable names (“Well, short names are quicker to type and might shave off 2% of our delivery time, but longer names will save us time if we need to revisit that part of the code…”) so there will always be ways to improve quality in refactoring, or configuration, or packaging, or architecture which the developer should just do without question because they’re a professional. But the developer who never informs their project board of major quality forks in the road, and who never allows their client to be involved in such decisions, is failing to fulfill their responsibility as a member of the whole project team.

At Guardian Unlimited we’re always looking for ways to improve the quality of what we do. But I’ve also heard people say “let’s not gold-plate it”, and “look, we’re not producing art”. These are senior developers who are arguing against particular quality improvements because they are able to weigh up not just the technical decisions but also the wider business decisions. They are being pragmatic. I’d say they are much better developers for it.

Non-executive summary for non-executives

Let’s recap. Quality is really important. It’s about much more than unit testing. And making small increments in quality can result in disproportionately large increments in the results. But let’s not kid ourselves that quality is a variable that can be tweaked arbitrarily and independently this way and that. It’s not of the same standing as budget or scope or time. And let’s not kid ourselves that developers of principle have to stand firm against some kind of management beast constantly trying to undermine them. Developers are an integral part of the organisation, and should behave (and be treated) as such. The quality of developers’ output needs to be balanced just as every other organisational demand does.

As long as we can see and understand software quality for what it really is, the better we are able to improve it.