There’s a terrific piece over on Computer Weekly by Alistair Maughan of law firm Morrison Foerster. He says the UK government’s embrace of agile is doomed to fail… so here I’m going to set out the case for the defence.

Actually, there’s much in Alistair’s text that I agree with, and it’s particularly heartening to hear a non-technologist speak intelligently about something many technologists have championed for so long. The same happened at the launch of the Institute for Government’s report which started this conversation in March, where many senior senior civil servants and political people spoke eloquently on the subject. But while Alistair does speak a lot of the truth, it’s not always the whole truth.

Alistair’s case is this:

The Cabinet Office expects Agile will reduce the risk of ICT project failure. It’s a nice idea in theory. But it won’t work in government IT. It won’t work in the real world. […] I’m prepared to accept on trust that, if all goes well, Agile may reduce the risk of project failure. But Agile simply won’t work in the real world of government ICT. But Agile simply won’t work in the real world of government ICT. We need a Richard Dawkins to bust the myth of the Agile gospel.

In fact he doesn’t need to take this on trust. In their report the IfG looked at several case studies from organisations (including mine) that really had used agile to reduce the risk of project failure — that’s actual evidence which “a Richard Dawkins” would well appreciate.

But, still, Alistair has set out a good case which is worth examining. It has four planks, so let’s take them in order…

1. Certainty of outcome

There are four clear reasons why Agile won’t work in government ICT. The most obvious is that government customers want to know up-front how much a system will cost. That’s not so unusual, is it?

Under Agile projects, you pay a given amount of money for a set amount of effort. But you can’t guarantee a specified outcome for a specific price.

This won’t work in government. Departmental budgets are managed very tightly, and they must be approved. Agile implies that charges for time & materials should be open ended. Government departments won’t accept that.

This is mostly true. Indeed, in part it’s more true than Alistair dwells on: not only are government department budgets managed very tightly, it’s common business responsibility. People want to know how what they’ll need to spend to get a known return. That’s not so unusual at all.

But it’s incorrect to say that charges for time and materials should be open-ended. That may be one way to structure a deal (and I bet plenty of suppliers would love that if they could get it) but it’s not the way it has to be. It is quite possible to govern an agile project with budgetary rigour. Estimation and planning are both part of the agile development process. A project can (and should) be set up with an overall business goal, plus key milestones along the way, with estimations between each point that both the customer (the government department) and the development team (the suppliers) sign up to. That is not open-ended. Agile does not give the development team a blank cheque or endless time — it requires their up-front commitment.

2. Procurement rules

Government is also legally required to follow open procurement rules.

That means comparing different bidders on a like-for-like basis, and deciding on best value for money. Agile can’t give you a clear specification of outputs up-front. Nor can it give a definitive [up-front] price.

So how are government bodies supposed to make Agile comply with the legal requirement that public procurements are fair and open?

Undoubtedly government procurement rules are a pain, particularly for small and medium-sized suppliers who might be well-suited to taking on agile (and by implication smaller) projects. But let’s put that aside, because the point here is not about bureaucracy, it’s about making like-for-like judgements.

Previously I said that a development team (whether in-house or an external supplier) — in conjunction with their customer — should be able to set out milestones and timescales which they can commit to. They may need to do a fair bit of work to understand what these are, but that would be the same for any approach to a large development project. In a waterfall-style project the “lot of work” would be a big up-front design based on hours of interviews and which was handed to the supplier (possibly with the aid of a fork lift truck); in an agile project the “lot of work” might consist of all-hands workshops and estimates and some spikes (i.e. actual uses of the technology to verify estimates), prior to structuring these into milestones. There are differences between these two approaches. Waterfall insists every detail is worked out up-front, which is unrealistic, while agile accepts there will be details and allows for them, even though they are not all determined up-front. Also, agile is more collaborative in its approach to planning. At the end you have a plan, with estimates, from each supplier and you can make a like-for-like judgement accordingly.

Another way to approach the “like-for-like problem” is to filter the suppliers based on price and ability. This is how we managed a competitive tender for our major rebuild and redesign project at the Guardian. We talked to a large number of potential partners, filtered it down to a small number who we were generally happy to work with, then spent additional time in workshops with each of them. Based on the outcome of those we made a final choice.

Our budget was much less than than a government department might have, but for the size of our organisation it was still significant so we had a responsibility to make a fair choice that would stand up to very high level scrutiny, and we achieved that.

3. Remedy

As if that isn’t problem enough, Agile offers insufficient means of remedy if things go wrong. […]

Agile makes it hard to apportion blame because the customer is intimately involved in the work. Since Agile contracts lack clear contractual delivery obligations or remedies, how do you enforce properly? How do you recover loss or damage if there’s a problem?

Once again, Alistair addresses a genuine concern which needs a solution.

Agile requires (or you might say “enables”) early and frequent delivery. This means that any problems should be identified early and remedies should be addressed as soon as problems arise. By spotting problems early any remedies should be lower cost. Significantly in an agile project, this means remedy options are much more numerous than in the picture that Alistair paints — a picture in which there is only one remedy, the nuclear option of legal and contractual finger-pointing. Ideally the search for an early remedy, following the early identification of a problem, will have an outcome which allows the project to continue to a satisfactory conclusion, rather than a failed project and one side compensated for said failure.

But early and frequent delivery is not the only solution here. As always, the client or customer cannot abdicate responsibility for good governance. If there are risks then they need to be addressed, perhaps by structuring the work accordingly. If there is a risk of your development partner not delivering to standard then perhaps the solution is to have two partners with similar responsibilities, so you can fall back on one if the other fails. Any particular project will have other risks, and there will be corresponding mitigation strategies.

You’ll notice, by the way, I’ve slipped my language from talking about a “supplier” to talking about a “partner”. The relationship really is different on an agile project, as Alistair acknowledges, but that doesn’t let anyone off the hook. The customer may be intimately involved in the work, but everyone should still have clear responsibilities set out, and they need to be accountable for delivering on those.

4. Hierarchy

Agile is fourthly not suited to public sector management structures.

Decision-making is centralised in government. […] It is inevitable that Agile decisions will go through management hierarchies in central government. This will be like kryptonite to Agile projects.

Agile projects rely on decisions based on mutual trust. They are therefore well suited to in-house projects. But the faith they ask customers to have in service providers makes them ill-suited for external developments.

Once more the answer here lies in governance, and this case appropriate governance. There is no doubt that if every decision has to go right the way to the top your agile project is doomed as it grinds to a near-halt. But equally if there is no means of sending any decisions to the top then the project is equally doomed as it has the risk of delivering the wrong thing very quickly.

People on the ground in an agile project need to be trusted, but strategic oversight and steering is still required on any large project. If it seems that milestones might be missed that has to be escalated (ideally with a constructive solution). If a partner isn’t performing senior people should know and make an informed judgement call. If it looks like business-level requirements might need to be adjusted it will need senior approval. Trust really is important, but blind trust is, shall we say, optimistic.

Accentuate the positive

Generally I will admit to being a glass-half-full person, and it pains me when I see naysayers try to halt an initiative which I have seen work elsewhere and which I believe can work in government. Adopting an agile approach for UK government IT is hugely exciting. I’ve visited one government department where they are trying agile and the individuals involved are genuinely excited about what they’re achieving — not what they’re hoping to achieve, but what they’re actually achieving.

This doesn’t mean merely adopting agile is a panacea. It’s not. As I’ve said above, it requires effective governance, and that’s a different kind of governance to what might be implemented on a more traditional waterfall-style project. And it’s inevitable there will be some mistakes along the way. No-one I know has adopted any new methodology without getting some things wrong and learning some lessons. That doesn’t mean agile is wrong for government.

The evidence is there that this works. This is not theory, this is practice and experience, and government needs to take that and make it their own. We shouldn’t bash government for taking this initiative — we must capture the early enthusiasm and help turn it into tangible success.