niksilver.com

The traditional way to track a project budget (which is to say, the way I learnt to do it first) is to track the time people spend on the project each day and add it up. I call this bottom-up budget tracking. Agile forces an alternative approach, which I call top-down. I was involved in a very happy example of this last week. Let’s talk first about how it works, then my experience this week, and finally a bit of a discussion.

The top-down approach simply requires you to cost your unit of measurement (ideal days, points, notional units of time, whatever) and then allows you to cost any any arbitrary grouping of functionality. Specifically, if your whole project costs C, and you’re on track to deliver W units of work, then one unit of work costs C/W. And if your sub-project totals P units of work then it will have cost you P * C/W.

The happy example I was involved in occurred when someone in our company unexpectedly decided we should know the cost of a particular subsection of our on-going redesign. It was 10 minutes’ work to find the stories which related to that work and convert story units into money. It was 20 minutes to explain to the visiting man-with-a-notebook how the numbers were derived, and the caveats.

In this particular case there were two caveats. One was that some planning and production work had taken place out of the scope of our project, so I couldn’t cost it. My man-with-a-notebook was fine with that because he already knew it was a cross-business piece of work and was already due to meet people in the other areas. The other caveat was that we were building on top of a large CMS, which clearly contributed to the value of the subsection he was interested in, but from which no cost could be reasonably apportioned. Again, he was fine with this because it was quite clear that the core CMS was built specifically for the main project while the additions we were discussing were very clearly for the subsection he was looking at. In the end he was just happy to have a clear financial figure.

The whole top-down approach is entirely dependent on the fact that your project is constantly delivering tangible features. This works on an Agile project because within each story you’ve bound up the analysis, architecture, development, QA and release. So the cost of each unit really does include everything. This doesn’t work for a waterfall-style project because although you may have only spent five days coding a particular feature, that also builds on some of your three weeks’ database schema design and implementation, some of your six weeks’ analysis time, some of your eight weeks’ testing time, and you almost certainly don’t have enough time tracking data to apportion the time correctly. But unlike the CMS in the example above you cannot reconcile the cost of the schema, analysis, etc to anything else because those things have no value or function without the delivery of the particular feature.

Naturally, though, there are pros and cons to the top-down approach.

The pros include the fact that it’s relatively easy to calculate. You may want to keep a running tally of how much a story unit costs — there will be good weeks and bad weeks — but overall it’s fairly straightforward, and you can do away with an awful lot of low-level accounting headaches.

Another benefit is that you can pull out any arbitrary group of stories and find their cost, as we’ve just seen.

A third benefit is that you know your numbers will add up: 100% of features delivered in the time will add up to 100% of your budget.

One of the major negatives I’ve found is in comparing it to the bottom-up approach. I remember a particular meeting with one project manager who was accounting bottom-up. His project intersected with an Agile one and he had been tasked with reporting on a particular cost, part of which was in the Agile project. As ever the bottom-up project manager had been asked for this report out of the blue. He met with the top-down, Agile project manager and me to get his data.

The top-down project manager wrote out a very simple calculation: the relevant work was, let us say, 3% of his overall project and the cost was therefore £24,500. However, the bottom-up project manager wasn’t comfortable with that: he wanted details. How much of that was PCs? How much software licences? How much time from developers? Consultants? Analysts…? There was some discussion, but there was no more detail to be had: the total project was £815,000, this was 3%, so the cost was £24,500. The top-down project manager explained his decision to keep administration to a minimum, explained again the cost structure of his project, and that’s all there was to it.

You can imagine the frustration felt there. The bottom-up project manager wanted to see the detail behind the cost, and he wasn’t comfortable with what he got. The detail he got was, “We delivered feature A for this much, feature B for this much, and feature C for this much.” The detail he wanted was names, hours, and asset numbers. The bottom-up project manager went away with concrete numbers but was dissatisfied, and I do wonder if he suspected his top-down colleague of financial recklessness. There were no repercussions from this, by the way. The financial report was submitted, accepted, and no more was heard. Nevertheless I would have liked a happier conversation.

Meanwhile the top-down project manager continued to report regularly to his project stakeholders, and that report naturally included budget status. The stakeholders were busy, senior people, who didn’t have the time or the desire to drill down into names, hours and asset numbers. For them a report saying that 42% of the functionality has been delivered with 45% of the budget was meaningful. They would query the big things, and the project manager needed to have detailed answers. A lightweight top-down approach allowed him to focus on those issues, and not get too distracted.

The names, hours and asset numbers are only enabling details, they are not ends in themselves. If they can possibly be avoided to focus on the more meaningful features then that’s a very good thing.

Mishkin Berteig says that rotating developers in and out of a support team should never be considered. I’d argue that he’s wrong, and that it’s often highly desirable and should always be considered.

In this article:

• Background. In which I outline what this is all about.
• Support rotation is desirable. In which I explain five reasons why rotating people through a support team is actually a good thing.
• Two asides. In which I get a little upset at the tone of some of some of Mishkin’s words (but only because I know he carries weight, you understand).
• When rotation is bad. In which I try to add a bit of balance.
• Summary. In which I round things off and try to make everybody happy.

So onwards…

Background

Mishkin Berteig writes about ways of handling software support — a much under-discussed issue in the public forums. But along the way he makes a comment I’d warn anyone to beware of:

A third common method is to form two separate teams: one doing new work, one doing support work. This is simple, effective, and annoying for the people on the team doing the support work! Please don’t consider a rotation system since this destroys the process of team development and makes it nearly impossible for the team doing new work to learn their capacity for the purposes of commitment.

I’d say the opposite: in many cases a rotation system is an excellent thing, is something to be strived for, and should be considered by anyone who needs to manage software support. Here’s why. I should say I’m going to blur the line between “support” and “maintenance” most of the time, here. I think that just reflects a lot of reality — support ranges from bug-fixing to minor improvements to not-so-minor improvements.

Support rotation is desirable

Let me count the ways…

1. Rotation win: Treating developers equally

Any manager wants all their developers to feel that they’re treated fairly. If support is considered different to project work (and it’s usually thought of as being less glamorous) then you won’t want a few of them consigned to the lesser work for all eternity. And they won’t want that either. Rotation into (and out of) the support team ensures everyone knows they have no greater or lesser privileges than anyone else when it comes to sharing the workload.

This issue becomes even more acute when we bring two other groups of people into the mix: contractors and new joiners. When new members or contractors join it’s so much easier to drop them into a nice, sandboxed project. They don’t need to know about all that long term legacy stuff that’s been around for ages and needs supporting. And so it becomes less likely that the longer-serving developers will ever escape the support team. Your longest serving developers — the very people who have stuck with you through thick and thin — will become those who feel most hard-done-by. Ensuring rotation through the support team evens all that out.

2. Rotation win: Maximising your team’s experience

Software development can’t be learned just by doing either project work or support work. You learn by doing both. The best developers talk not just about creating new code, but also about maintainability, and will do so from experience. Just from a general perspective people need a breadth of experience.

But that’s important, too, even at a more specific level. It’s no good writing a piece of software and they saying adios the moment it’s launched. An application arguably only begins its life when it’s released. That’s when it starts to be really used, when it meets its users, and that’s when the lessons are to be learned.

Oh, that class really doesn’t make sense after all… That design for the whizzbat component really isn’t doing us any favours… Thank goodness we designed the look-up tables as we did, we should remember to do that again.

I’ve said to individual developers before, and I’m sure I’ll say it again, the codebase has to be in a better state after you’ve touched it than before. That skill is really best developed in a support team. A developer who doesn’t experience the full lifecycle — either in general or of a particular piece of software — isn’t learning as much as they could. Rotation through the support team ensures a broader experience and better insight into their trade.

3. Rotation win: Knowledge sharing

Fascinating that Mishkin advises against rotation “since this destroys the process of team development”. I’m not sure if he’s referring to developers’ general experience or to learning about the particular software that’s coming together in the project. If he’s talking about the former then I’ve covered that above. But if he’s talking about the latter point, then team rotation is an excellent way to ensure it.

Of course rotation can happen within a team to ensure knowledge sharing. If you do pair programming, then pair rotation promotes knowledge sharing within the team. But more than that, rotation between teams shares knowledge even more widely. If you’re going to do frequent releases then that release suddenly goes into maintenance mode and needs to be supported. If you can rotate a project team member into the support team then the support team as a whole suddenly gains knowledge of the software, they can live up to their new responsibility, and individual team members can learn from the incoming developer.

4. Rotation win: Maximising internal prospects

There’s a corollary to rotating developers from the project team into the support team throughout the life of the project. And that’s that it prevents the situation where the project team completes the project and then stays with it for life because they’re the only ones who understand it. In that situation the team as whole changes from a project team to a support team, and worse than that, they have no variety and become the support team of just a single product. By constantly refreshing the project team you ensure that developers can see that their prospects within the organisation are maximised.

5. Rotation win: Facing up to difficult issues

The issues above are mainly benefits to developers. But there’s a benefit which is solely for the manager and the business as a whole: aiming for rotation through the support team forces the manager to face up to difficult issues, whether that’s maintainability, team motivation, or knowledge sharing. Support rotation is a concrete and visible goal. By aiming for that goal you’re aiming to confront those issues, and as long as you’ve not (visibly) achieved it you’ve not dealt with the issues. Rotation through the support team provides a visible mark of success.

A couple of asides

Two things in passing. First, I think of the knowledge sharing issue almost as an extension of extreme programming. The principle in XP is, roughly, take all the good things and turn the dial up to eleven. To me, knowledge sharing is a good thing, and rotation between teams is one way to turn that dial right up.

Second — and I’m really reluctant to raise this, but think it’s important — we’ve got to be very careful about being prescriptive. The throwaway phrase warning against rotation (”Please don’t consider a rotation system since this destroys the process of team development and makes it nearly impossible for the team doing new work to learn their capacity for the purposes of commitment”) has too many bad echoes for me of the “you call this agile?” ding-dong.

You may recall that also began with a remark posted on the Agile Advice blog. There, the author seemed to categorically refuse any interruptions to his developers’ week on the grounds that two hours’ interruption can waste the entire week. Joel O’Software countered this, suggesting that maybe, just maybe, an interruption might be okay. There were arguments both ways [1, 2, 3, etc]. In both cases — on rotation and interruptions — the impression is created that developers are revered saints who need near-religious attendance or they’ll fall to pieces. The best developers I know have their feet planted firmly on the ground, and the best businesses I know treat developers with respect not because they’re developers, but because everyone there treats everyone else with respect. It’s very rarely black-and-white when methodologies meet reality. We need to be pragmatic, not dogmatic, and we need to be seen as such.

Thank you for indulging me. Now back to our regular programming…

When rotation is bad

Yes, yes, one mustn’t be dogmatic. Rotation in and out of the support team won’t always work…

Caution: It’s disruptive

Rotation is disruptive, for exactly the reasons that Mishkin gives, so it’s got to be balanced against everything else that’s going on in the business. Rotating a developer every day is clearly going to be a bit much. But rotation can be planned to minimise disruption. The point when one iteration stops and the next starts is obviously a good opportunity. Maybe not between every iteration, but at appropriate project milestones, when people return from holiday, and so on.

Caution: It depends what “support” means

In all the above I’ve really thought of “support” as being almost synonymous with “development on a very small scale”. But it’s not always like that. In many cases software support involves telephone support, or perhaps meeting paying customers. Clearly this requires a very different (interpersonal) skillset than you’d look for otherwise in a software developer. Many developers don’t want to spend time meeting besuited customers, and in these cases you’ll end up hiring different kinds of people for different kinds of teams. Clearly here rotation isn’t going to work.

On the other hand I should say I know a few developers who do like meeting customers, and who do like solving problems on the phone. And I also know developers who like maintenance work without meeting customers because it gives them a strong sense of satisfaction to see a closed problem solved in a timespan that’s a fraction of most projects.

Caution: If you’ve already committed

There’s also one final occasion when support rotation is difficult, and that’s when you’ve promised people otherwise. Perhaps you’ve undertaken to a new recruit that “no, we won’t put you on support”. Or perhaps you’ve promised a particular client that certain individuals will stay on their project to the end. In these cases, obviously, you’ve got to honour your word.

Summary

Support is a very, very under-discussed topic, particularly in the field of Agile development where the writing tends to be about projects rather than maintenance. So Mishkin Berteig has done us all a great service by spending some time on it, and offering suggestions about how it might be managed. But there are probably more ways than even he suspects, and in particular — and against his advice — I would very strongly recommend giving serious consideration to rotating developers in and out of a support team.

It’s not easy to manage, and there reasons not to do it. But there are also plenty of very good reasons that it should be done, and hopefully the words above will encourage its wider adoption.

We work using agile development processes, which obviously needs the buy-in of internal users and project sponsors. But this jumped out at me when I read it. It’s from Carolyn McCall, Chief Executive of Guardian Media Group, which owns the company I work for. She was announcing a £15m investment in our digital business, and it was reported thus:

“What we’ve done so far is our own version of web 1.0, but we want to continue to web 2.0 and what comes after that,” Ms McCall added. “We need to be agile and ready to change.”

I suspect that phrase, “We need to be agile and ready to change”, is not a coincidence. You can trace a strong lineage between the agile development work going on in Guardian Unlimited and that little sentence in a speech to the OPA. Let me tell you how I think it got there.

A story of a sentence

A long time ago we began using agile processes in the GU tech team. It was a new way of working, but it seemed to address a lot of our problems — actually, the usual ones you get in most software organisations, such as knowledge silos, lack of flexibility, impossible deadlines, and so on. Changing what you know is always a tough choice, but the GU management team were very supportive of the move to agile.

In particular, our boss at GU at the time, Simon Waldman, was very interested in its implementation and evolution. (Simon’s since moved to GMG.) Over the months and years he enquired, queried, provoked, but was never less than encouraging and constructive. (At one memorable annual review to the GU staff Simon devoted an entire slide to explain how we had stealthily removed a problematic database table. Our DBA was thrilled; I have no idea what the attendant sales people and journalists made of it.)

And agile clearly worked for GU. One notable change was that planning meetings took on much more of a business focus. Far less about technical dependencies, much more on what functionality we wanted to release when.

And then we began we work on our new Travel site (released in November). It featured a new design and new commercial opportunties. More people around the company needed to scrutinise this as it started up, because it was a pretty big project and we had a much greater responsibility to be rigorous and transparent in our delivery. We saw agile methods as being critical to the work’s success. It would allow us to delay decisions to the latest possible moment and therefore produce something that was much more relevant — both editorially and commercially — to our audience and clients.

I therefore spent some time explaining agile to various company directors… but not as much as you might expect, because clearly the principles and practices had been discussed and understood well before. I ended up having the kind of conversations about agile development with non-technical senior managers that I wish I could have had with more technical job candidates. Word was getting around. The business people scrutinising the work on the Travel site understood the process’s business benefits, and they understood how the low-level practices would provide those, and they knew those practices were working successfully within GU at the time. And indeed, the project was blessed by the the company directors, including Simon and our MD at the time… Carolyn McCall.

You can see how that word “agile” has been thrown around a lot inside the company, and was — at least then — strongly associated with Guardian Unlimited and therefore its innovation (which I’d like to think is almost synonymous with “daily work” — ahem), and it’s stuck.

Now I can’t claim that Carolyn knows what, say, test-driven development really is. I suspect she has higher level things to think about. But if it came up, I think she’d understand our practices such as refactoring (making lots of tiny internal restructurings to produce a smoother, more manageable system), continuous integration (ensuring our changes are constantly integrated into the day-to-day work, not siloed) and retrospectives (frequent reviews and actions to improve). And, come to think of it, TDD, which is really about supporting and enabling change.

It seems Carolyn understands “agile” in a way that I wish more technical people would. It’s about change, and about supporting change (”We need to be agile and ready to change”).

The hard work starts now

And what of the future? Well, agile development helps us deliver responsibly to the business — and if someone invests £15m in you, then you really need to deliver responsibily. Again, that kind of investment only happens if there’s confidence in your ability to deliver. Can we offer that confidence? Well, just this week I was in a meeting with Tom Turcan, who is our Head of Digital Media Development. The conversation went something like this. Note that development for this particular thing is already about 30% of the way through:

Expert user: “…So in summary we’ve decided that features A, B and C aren’t that important after all, and we’ve replaced them with features J and K.”

Tom: “It looks like A, B, C total 7 ideal days, and J and K total 8.5 ideal days. So that’s a net increase of 1.5 ideal days. Is that right?”

Expert user: “Er, yes.”

Tom: “So we can’t really allow that to happen. We can make changes to make the work less, or keep it at the same level, but we mustn’t start increasing it. Can you find 1.5 days’ worth of features to remove?”

Expert user: “Yes, I should think so.”

Tom: “Okay, we’ll make a note to see what you’ve decided next week.”

We couldn’t have had that conversation in a more waterfall environment. By the time we’d have reached this point the groundwork for A, B and C — the database tables, the DAO layer, etc — would have been done, and it would have been wasted effort. Instead our expert user is able to make decisions later, and Tom is able keep to keep the work under control, giving the rest of the business the confidence it needs. Our agile processes enable that to happen. And all that is known by our developers, and by our managers, and that same confidence is shared by our group Chief Executive.

Postscript

By the way, there are other things going on this week in and around Guardian Unlimited. You might want to take a look at Jeff Jarvis’s commentary on Alan Rusbridger’s commitment to digital and his interest in being flexible. Also, my esteemed colleague Neil McIntosh tells how GU saved a cat, and offers a word of advice to a researcher at News International.

As a friend said to me the other day, “I know when you’re busy at work — your blog goes quiet.” So back to work now.