niksilver.com

The traditional way to track a project budget (which is to say, the way I learnt to do it first) is to track the time people spend on the project each day and add it up. I call this bottom-up budget tracking. Agile forces an alternative approach, which I call top-down. I was involved in a very happy example of this last week. Let’s talk first about how it works, then my experience this week, and finally a bit of a discussion.

The top-down approach simply requires you to cost your unit of measurement (ideal days, points, notional units of time, whatever) and then allows you to cost any any arbitrary grouping of functionality. Specifically, if your whole project costs C, and you’re on track to deliver W units of work, then one unit of work costs C/W. And if your sub-project totals P units of work then it will have cost you P * C/W.

The happy example I was involved in occurred when someone in our company unexpectedly decided we should know the cost of a particular subsection of our on-going redesign. It was 10 minutes’ work to find the stories which related to that work and convert story units into money. It was 20 minutes to explain to the visiting man-with-a-notebook how the numbers were derived, and the caveats.

In this particular case there were two caveats. One was that some planning and production work had taken place out of the scope of our project, so I couldn’t cost it. My man-with-a-notebook was fine with that because he already knew it was a cross-business piece of work and was already due to meet people in the other areas. The other caveat was that we were building on top of a large CMS, which clearly contributed to the value of the subsection he was interested in, but from which no cost could be reasonably apportioned. Again, he was fine with this because it was quite clear that the core CMS was built specifically for the main project while the additions we were discussing were very clearly for the subsection he was looking at. In the end he was just happy to have a clear financial figure.

The whole top-down approach is entirely dependent on the fact that your project is constantly delivering tangible features. This works on an Agile project because within each story you’ve bound up the analysis, architecture, development, QA and release. So the cost of each unit really does include everything. This doesn’t work for a waterfall-style project because although you may have only spent five days coding a particular feature, that also builds on some of your three weeks’ database schema design and implementation, some of your six weeks’ analysis time, some of your eight weeks’ testing time, and you almost certainly don’t have enough time tracking data to apportion the time correctly. But unlike the CMS in the example above you cannot reconcile the cost of the schema, analysis, etc to anything else because those things have no value or function without the delivery of the particular feature.

Naturally, though, there are pros and cons to the top-down approach.

The pros include the fact that it’s relatively easy to calculate. You may want to keep a running tally of how much a story unit costs — there will be good weeks and bad weeks — but overall it’s fairly straightforward, and you can do away with an awful lot of low-level accounting headaches.

Another benefit is that you can pull out any arbitrary group of stories and find their cost, as we’ve just seen.

A third benefit is that you know your numbers will add up: 100% of features delivered in the time will add up to 100% of your budget.

One of the major negatives I’ve found is in comparing it to the bottom-up approach. I remember a particular meeting with one project manager who was accounting bottom-up. His project intersected with an Agile one and he had been tasked with reporting on a particular cost, part of which was in the Agile project. As ever the bottom-up project manager had been asked for this report out of the blue. He met with the top-down, Agile project manager and me to get his data.

The top-down project manager wrote out a very simple calculation: the relevant work was, let us say, 3% of his overall project and the cost was therefore £24,500. However, the bottom-up project manager wasn’t comfortable with that: he wanted details. How much of that was PCs? How much software licences? How much time from developers? Consultants? Analysts…? There was some discussion, but there was no more detail to be had: the total project was £815,000, this was 3%, so the cost was £24,500. The top-down project manager explained his decision to keep administration to a minimum, explained again the cost structure of his project, and that’s all there was to it.

You can imagine the frustration felt there. The bottom-up project manager wanted to see the detail behind the cost, and he wasn’t comfortable with what he got. The detail he got was, “We delivered feature A for this much, feature B for this much, and feature C for this much.” The detail he wanted was names, hours, and asset numbers. The bottom-up project manager went away with concrete numbers but was dissatisfied, and I do wonder if he suspected his top-down colleague of financial recklessness. There were no repercussions from this, by the way. The financial report was submitted, accepted, and no more was heard. Nevertheless I would have liked a happier conversation.

Meanwhile the top-down project manager continued to report regularly to his project stakeholders, and that report naturally included budget status. The stakeholders were busy, senior people, who didn’t have the time or the desire to drill down into names, hours and asset numbers. For them a report saying that 42% of the functionality has been delivered with 45% of the budget was meaningful. They would query the big things, and the project manager needed to have detailed answers. A lightweight top-down approach allowed him to focus on those issues, and not get too distracted.

The names, hours and asset numbers are only enabling details, they are not ends in themselves. If they can possibly be avoided to focus on the more meaningful features then that’s a very good thing.

I was surprised to see Ken Schwaber talking about burn-down charts, as burn-up charts provide more information and are — for me — the preferred option. So this is a short article about burn-up charts and burn-down charts, both great tools for measuring progress.

First, though, a hat-tip to Ian Carroll, a colleague at MEN Media. His blog shows off the video I saw. In it, Ken explains Scrum and answers a few questions from Google employees. It was worth an hour of my time just for his explanation of the wider consequences of dropping quality. The argument may be familiar to you, but the clarity of the explanation and the use of the visual aid (a burn-down chart) is terrific. If you want to see just that then jump to minute 40.

Burn-down charts

A burn-down chart tracks how much work remains on your project and whether you’ll hit your deadline. The vertical axis measures work remaining. The horizontal axis marks your iterations, and you should mark which iteration is your target end date for the project. After each iteration you mark your progress on the chart and you can project forward to see whether or not you’ll hit your target end date.

In the diagram above we’ve gone four iterations, and the dotted line (the projection) suggests we’ve fallen quite badly behind schedule. After the first iteration you can see we were making good progress, but things changed in the second iteration and alarm bells should have sounded. The project manager should have started doing something then.

Burn-up charts

A burn-up chart tracks how much work is done. But it shows more information than a burn-down chart because it also has a line showing how much work is in the project as whole (the scope as workload), and this can change. On the burn-down chart it is harder to show a changing goal.

In the diagram the bottom line is the work done and the top line is the total scope. Dotted lines show projections. We can see that the scope remained steady for the first few iterations, and we would have expected to make the deadline. But scope increased around the fifth iteration. Even this wouldn’t have been too bad if that had been the end of it, but it continued to increase after that. The projections show that we won’t make the target deadline at our current rate of scope creep. In fact we probably won’t even make it if the scope remains steady.

Getting advanced

More sophisticated burn-up charts can show several scope lines, one for each release milestone. This way the team and the stakeholders can see the effects of, say, moving scope from one release to another.

Burn-down charts have the advantage that they’re simpler — they only show one line. Perhaps that’s why Ken used them in the video; much more would have distracted from his messages. But if you can stand a little complexity then I’d recommend burn-up charts as the more useful option for real-world situations.

When do we need user acceptance testing? And when can we get away without it?

User acceptance testing (UAT) is when your software goes in front of the user to get final sign-off — and when they ask for changes if not. In theory you shouldn’t need UAT at all (didn’t they tell you what they wanted? Weren’t you listening?), and indeed perhaps you can sometimes get away without it.

This argument is stronger if you’re considering UAT throughout the project, not just at the end. This would be the case if you’re UATing individual features, or individual deliveries in a culture of more and smaller iterations. The arguments are: more deliverables mean more UAT means more expense for the customer; smaller iterations mean a reduced chance of getting it wrong; frequent deliverables give the customer more opportunities to change anything they don’t like anyway.

However, even with small, frequent deliveries there are circumstances when it’s more important to do UAT. Looking back on some projects I’ve worked on, here are times when it was, or would have been, a good idea…

1. When the output of the software is subjective

I once worked on a project that asked what bands you liked and then based on that recommended albums by unsigned bands. This was a big leap away from the “people who liked that also liked this…” kind of recommendations, because the unsigned bands won’t have had a significant userbase, so we couldn’t rely on a self-generating wisdom-of-crowds mechanism. Plus, of course, music recommendation is highly subjective, and it was our clients who were the music experts — we implementers were mere software developers.

In this case you can see it was key for our clients to play with the system and be sure they were comfortable with it. If they didn’t then it would have meant tweaking the algorithm, not a complete software rewrite of the software. In the event something rather unexpected happened. While our client’s representative was very happy with the output he also found it rather disconcerting. He felt somewhat disenfranchised from the system because it seemed so mysterious. He realised that he couldn’t demonstrate it in front of his colleagues and their investors without having an answer for the suddenly inevitable question: how does it work? And once this was explained to hiim he became much more comfortable again.

If the output of the system had been much more objective and predictable this instance of UAT wouldn’t have been so important. In the end it threw up an important new acceptance criteria for client which we were quickly able to address.

2. When the software has a costly userbase

One of my past projects involved creating a user interface for a property data entry team. Individuals on the team were, as you’d expect, low-paid and largely interchangeable — after an hour of training you would know everything there was to know about the system. Your sole function was then to spend hour after unsociable hour entering information about properties you could never afford to live in for people you would never want to live next to.

But although the daily cost of individuals on the team was considered low, the cost of the team as a whole — and the value they were bringing to the business — was very high.

Problems with the user interface in this case would have had a knock-on effect across the whole team of 20 or 30 individuals, and cost their employer dear. It was key for someone to check not only that the system was responsive, but also that there were intuitive keystrokes across the data fields, data entry was reasonably forgiving, tabbing between fields happened in the order that data was presented in, and so on. In this case, the time and early feedback from one user at the vanguard saved time and cost 20 or 30 times over.

3. When you don’t have a QA department

A pretty obvious one, but if you don’t have people dedicated to quality assurance then those internal people who take on that function (probably the lucky developers or irritated account manager) will be too close to the software and too distant from client to pick up on all the issues that are important to them.

A good percentage of the projects I’ve worked on have involved integrating search engines. And in almost every instance when testing has been left to the development team they’ve alighted on one or two key phrases to use as test cases and considered that sufficient. By contrast, the end user to whom this is important will have half a dozen phrases that are relevant each day — and they’ll involve accented characters, non-ASCII characters, mismatched quote marks, and so on. And that’s before we get to the results page. Without effort the expert user has given the search engine a good workout — often raising difficult issues for the developers.

Actually, even if you do have a QA department then there’s a case for UAT, because more often than not the QA team will test against written criteria. There are very often criteria which, with the best effort in the world, just don’t get realised or written down at the requirements stage. And in these cases it’s only the end user who can really tell if it’s right.

4. When you lack detailed requirements

Another fairly obvious situation, but if you’ve not had a good bit of customer input at the start of the project, then UAT is going to be even more important later on.

A sales person I worked with once had a cunning plan. We’d already produced an e-commerce site for one of our clients, and he decided to sell a duplicate system to one of their rivals. It was good plan on paper. There was no contractual restriction for us; the new client would get their site at a relatively low cost; they’d get it quicker than usual; and we’d only have to reskin it. The deal was done over a nice lunch.

You can guess what happened in reality. When they saw what we imagined was a near-complete version they wanted some changes. The checkout system wasn’t quite to their liking; their product catalogue didn’t have quite the fields we were expecting; the order placement system didn’t align with what they had internally; and so on. It cost everyone more than they had anticipated and it was delivered later than anyone would have liked.

Certainly this is a strong case for robust specifications — or at least understanding what you’re getting into before you get into it. But it also demonstrates that lack of early information cost us dearly when the information did come along later. We hadn’t made time for UAT, which is why we delivered the final product later than planned. Fortunately for everyone there was no externally-driven deadline. If there had been then we’d have been obliged to deliver something which the client wasn’t happy with.

5. When the inputs to the software are difficult to specify

Finally an example that brings us right to the present.

The current work to refresh the Guardian Unlimited website is much more than applying a lick of paint. We’re also producing new tools for our reporters and new ways they can tell their stories. But telling a news story is not simply a matter of filling in a few boxes and clicking Save. To make a story relevant requires a lot of hard training. To edit an entire news section additionally needs an awful lot of experience and good intuition…

As I write, MediaGuardian.co.uk is leading with three big stories and a podcast, plus some smaller items headed “More media news”. The stories have between three and six sublinks which indicates the depth of related news that the editor has available. Yet at the same time the Technology section is showcasing six stories, all with a bit of blurb, but only one of which has sublinks (to a video and a gallery). In both cases the editors are using the same system, but the page layout has to balance itself out regardless of the highly variable input.

But though the input is highly variable it is not without bounds. A section would never be showing just one story, and an editor would almost certainly not try to highlight twenty stories and claim they had largely equal weight. The only way to really know the bounds in which the system is supposed to reasonably work is to sit an editor down with the tools and let them lay out pages in response to a real day’s stories. No amount of requirements analysis and QA experience can substitute for a real journalist responding to a real day’s events.

And that’s the beginning…

No mechanism should be applied blindly; there are times when user acceptance testing is more appropriate, and times when it is less appropriate. The important thing is to be aware first that UAT exists, and second that there are criteria which you can apply to assess how valuable it can be.