Last week I was pointed to a report by National Audit Office called Governance for Agile delivery. It’s a really terrific piece of writing and I’d recommend it to anyone … Continue reading Agile governance from the National Audit Office
I spent a couple of days this week hearing from Tom Gilb about defect prevention. Here is just one small thing from that which intrigued me… rules for business cases. … Continue reading Rules for business cases
A thumbleful of notes from a couple of the sessions of the Agile Business Conference 2011. First Jerrett Myers’ and James Yoxall’s talk on lessons implementing an agile project for … Continue reading Serious agile: Agile Business Conference 2011
Can we reasonably perform programme management as a form of portfolio management? This was the question I discussed with some peers earlier this week. On the surface this an attractive … Continue reading Programme, portfolio and asset management
There’s a terrific piece over on Computer Weekly by Alistair Maughan of law firm Morrison Foerster. He says the UK government’s embrace of agile is doomed to fail… so here I’m going to set out the case for the defence.
Actually, there’s much in Alistair’s text that I agree with, and it’s particularly heartening to hear a non-technologist speak intelligently about something many technologists have championed for so long. The same happened at the launch of the Institute for Government’s report which started this conversation in March, where many senior senior civil servants and political people spoke eloquently on the subject. But while Alistair does speak a lot of the truth, it’s not always the whole truth.
Alistair’s case is this:
The Cabinet Office expects Agile will reduce the risk of ICT project failure. It’s a nice idea in theory. But it won’t work in government IT. It won’t work in the real world. […] I’m prepared to accept on trust that, if all goes well, Agile may reduce the risk of project failure. But Agile simply won’t work in the real world of government ICT. But Agile simply won’t work in the real world of government ICT. We need a Richard Dawkins to bust the myth of the Agile gospel.
In fact he doesn’t need to take this on trust. In their report the IfG looked at several case studies from organisations (including mine) that really had used agile to reduce the risk of project failure — that’s actual evidence which “a Richard Dawkins” would well appreciate.
But, still, Alistair has set out a good case which is worth examining. It has four planks, so let’s take them in order…
1. Certainty of outcome
There are four clear reasons why Agile won’t work in government ICT. The most obvious is that government customers want to know up-front how much a system will cost. That’s not so unusual, is it?
Under Agile projects, you pay a given amount of money for a set amount of effort. But you can’t guarantee a specified outcome for a specific price.
This won’t work in government. Departmental budgets are managed very tightly, and they must be approved. Agile implies that charges for time & materials should be open ended. Government departments won’t accept that.
This is mostly true. Indeed, in part it’s more true than Alistair dwells on: not only are government department budgets managed very tightly, it’s common business responsibility. People want to know how what they’ll need to spend to get a known return. That’s not so unusual at all.
But it’s incorrect to say that charges for time and materials should be open-ended. That may be one way to structure a deal (and I bet plenty of suppliers would love that if they could get it) but it’s not the way it has to be. It is quite possible to govern an agile project with budgetary rigour. Estimation and planning are both part of the agile development process. A project can (and should) be set up with an overall business goal, plus key milestones along the way, with estimations between each point that both the customer (the government department) and the development team (the suppliers) sign up to. That is not open-ended. Agile does not give the development team a blank cheque or endless time — it requires their up-front commitment.
2. Procurement rules
Government is also legally required to follow open procurement rules.
That means comparing different bidders on a like-for-like basis, and deciding on best value for money. Agile can’t give you a clear specification of outputs up-front. Nor can it give a definitive [up-front] price.
So how are government bodies supposed to make Agile comply with the legal requirement that public procurements are fair and open?
Undoubtedly government procurement rules are a pain, particularly for small and medium-sized suppliers who might be well-suited to taking on agile (and by implication smaller) projects. But let’s put that aside, because the point here is not about bureaucracy, it’s about making like-for-like judgements.
Previously I said that a development team (whether in-house or an external supplier) — in conjunction with their customer — should be able to set out milestones and timescales which they can commit to. They may need to do a fair bit of work to understand what these are, but that would be the same for any approach to a large development project. In a waterfall-style project the “lot of work” would be a big up-front design based on hours of interviews and which was handed to the supplier (possibly with the aid of a fork lift truck); in an agile project the “lot of work” might consist of all-hands workshops and estimates and some spikes (i.e. actual uses of the technology to verify estimates), prior to structuring these into milestones. There are differences between these two approaches. Waterfall insists every detail is worked out up-front, which is unrealistic, while agile accepts there will be details and allows for them, even though they are not all determined up-front. Also, agile is more collaborative in its approach to planning. At the end you have a plan, with estimates, from each supplier and you can make a like-for-like judgement accordingly.
Another way to approach the “like-for-like problem” is to filter the suppliers based on price and ability. This is how we managed a competitive tender for our major rebuild and redesign project at the Guardian. We talked to a large number of potential partners, filtered it down to a small number who we were generally happy to work with, then spent additional time in workshops with each of them. Based on the outcome of those we made a final choice.
Our budget was much less than than a government department might have, but for the size of our organisation it was still significant so we had a responsibility to make a fair choice that would stand up to very high level scrutiny, and we achieved that.
As if that isn’t problem enough, Agile offers insufficient means of remedy if things go wrong. […]
Agile makes it hard to apportion blame because the customer is intimately involved in the work. Since Agile contracts lack clear contractual delivery obligations or remedies, how do you enforce properly? How do you recover loss or damage if there’s a problem?
Once again, Alistair addresses a genuine concern which needs a solution.
Agile requires (or you might say “enables”) early and frequent delivery. This means that any problems should be identified early and remedies should be addressed as soon as problems arise. By spotting problems early any remedies should be lower cost. Significantly in an agile project, this means remedy options are much more numerous than in the picture that Alistair paints — a picture in which there is only one remedy, the nuclear option of legal and contractual finger-pointing. Ideally the search for an early remedy, following the early identification of a problem, will have an outcome which allows the project to continue to a satisfactory conclusion, rather than a failed project and one side compensated for said failure.
But early and frequent delivery is not the only solution here. As always, the client or customer cannot abdicate responsibility for good governance. If there are risks then they need to be addressed, perhaps by structuring the work accordingly. If there is a risk of your development partner not delivering to standard then perhaps the solution is to have two partners with similar responsibilities, so you can fall back on one if the other fails. Any particular project will have other risks, and there will be corresponding mitigation strategies.
You’ll notice, by the way, I’ve slipped my language from talking about a “supplier” to talking about a “partner”. The relationship really is different on an agile project, as Alistair acknowledges, but that doesn’t let anyone off the hook. The customer may be intimately involved in the work, but everyone should still have clear responsibilities set out, and they need to be accountable for delivering on those.
Agile is fourthly not suited to public sector management structures.
Decision-making is centralised in government. […] It is inevitable that Agile decisions will go through management hierarchies in central government. This will be like kryptonite to Agile projects.
Agile projects rely on decisions based on mutual trust. They are therefore well suited to in-house projects. But the faith they ask customers to have in service providers makes them ill-suited for external developments.
Once more the answer here lies in governance, and this case appropriate governance. There is no doubt that if every decision has to go right the way to the top your agile project is doomed as it grinds to a near-halt. But equally if there is no means of sending any decisions to the top then the project is equally doomed as it has the risk of delivering the wrong thing very quickly.
People on the ground in an agile project need to be trusted, but strategic oversight and steering is still required on any large project. If it seems that milestones might be missed that has to be escalated (ideally with a constructive solution). If a partner isn’t performing senior people should know and make an informed judgement call. If it looks like business-level requirements might need to be adjusted it will need senior approval. Trust really is important, but blind trust is, shall we say, optimistic.
Accentuate the positive
Generally I will admit to being a glass-half-full person, and it pains me when I see naysayers try to halt an initiative which I have seen work elsewhere and which I believe can work in government. Adopting an agile approach for UK government IT is hugely exciting. I’ve visited one government department where they are trying agile and the individuals involved are genuinely excited about what they’re achieving — not what they’re hoping to achieve, but what they’re actually achieving.
This doesn’t mean merely adopting agile is a panacea. It’s not. As I’ve said above, it requires effective governance, and that’s a different kind of governance to what might be implemented on a more traditional waterfall-style project. And it’s inevitable there will be some mistakes along the way. No-one I know has adopted any new methodology without getting some things wrong and learning some lessons. That doesn’t mean agile is wrong for government.
The evidence is there that this works. This is not theory, this is practice and experience, and government needs to take that and make it their own. We shouldn’t bash government for taking this initiative — we must capture the early enthusiasm and help turn it into tangible success.
A few random notes on the role of governance in decentralised organisations. This comes from a discussion I took part in earlier today with peers from a range of organisations.
We agreed that governance is about (a) surfacing pertinent issues and (b) helping people make decisions. But exactly what needs to be surfaced depends on what people wants to know. It may be
- Confirming regulatory compliance;
- Confirming the stakeholders are happy (a real example of one participant);
- Ensuring projects are moving rapidly;
- Measuring operational progress against predefined KPIs;
- …and so on.
What you want to surface, and what decisions the senior management team will want to take, determines what information gets collected and presented.
It was acknowledged that governance processes can be very prescriptive to people in projects — for example determining test procedures or release processes — but we generally saw that as a bad thing. In particular, it is rare that one procedure is the best fit for every project. It’s harder for that to happen in a decentralised organisation, but not impossible.
One participant outlined his ideal governance model: At various points throughout a project’s lifecycle a check is made against all of a number of criteria. For example, requirements coverage, security, quality, compliance, commercial model, etc. For each criterion only an authoritative person can say it is satisfactory. So only the head of security or one of her representatives can confirm the security is appropriate; the project manager cannot confirm her approval second hand. If something is not confirmed the oversight group flags it. Notably, they do not fix it, nor do they mandate any actions. It is up to the senior management group to take any actions where it sees fit. Thus the oversight group does not control what people do, and authority continues to be devolved.
What I’m left thinking about now is this: Governance is about assurance and decision support. If the governance process is too light it is pointless. If it is too heavy it adds weight and slows things down. Now, can you be sure you’ve got it right…?
I’ve been thinking about goal-setting — partly in the process of writing my last blog post, and partly because I’ve been reading about the Beyond Budgeting approach to management. There are plenty of thought-provoking ideas in Jeremy Hope and Robin Fraser’s book, but I want to throw out just one of the smaller ones which caught my attention: the idea of using ratios as KPIs, targets, and limits. These seem a really effective way of getting things done.
The kinds of targets that I hear about are very often single-unit targets: We want to have 200,000 subscribers; We want to break 3m monthly unique browsers; We want to get to 1m comments on the website. These are measured in single units: subscribers, monthly unique browsers, comments.
But perhaps more useful are ratio targets. Hope and Fraser have a financial perspective and talk about return on capital, cost-to-income ratio, and cost per head. There are also some standard ratio measures in the industry I work in, such as average revenue per user (ARPU). But we’re not exploiting this kind of thing enough, and such measures don’t have to be industry standard — it’s often more useful if they’re specific to an organisation.
Let’s try to imagine some more. Take the example of Quora, which has just hired product lead Sandra Liu Huang from Facebook. What kind of measurements might we consider for her as she enters her new world? We might have cost to serve per unique user; answers per question; questions answered per user; updates per user per month; and plenty of others.
Why they make a difference
What really makes a difference with these kinds of measures is that they provide people with more levers to make success happen:
- Seeking to reduce cost to serve per unique user offers the chance to reduce the costs of our server infrastructure, or increase (presumably) the number of unique users, or (much smarter) manage our server infrastructure so that it costs less to flex it in response to changes in usage.
- Seeking to increase answers per question suggests we might look at the user interface to emphasise answering, or make it more difficult for people to ask silly questions.
- …and so on.
If people have more levers then they have more opportunities to make success happen. That means the measures can’t just be kept inside the finance department or put in the shareholders’ report — they must be exposed and explained to everyone who contributes. Then everyone in the organisation can work together.
And this needn’t just be about relative improvements. In the bullets above I’ve used the metrics in relative improvement targets (decrease this, increase that) but they could equally be used in absolute targets (we want this figure to be X) or limits (make sure this number doesn’t fall below Y).
Mixing financial and non-financial numbers
Picking up from my last post about innovation from trust, this becomes very powerful when we mix financial and non-financial units. So previously the board might have set a goal to “increase our userbase to 2.5m users by the end of the financial year”, and (if they were really organised) might also have set aside, say, £300k capital to achieve it. Unfortunately that doesn’t offer the team (the people who have to deliver) much opportunity to make it happen, because (a) it’s very unlikely that in the rush of the annual budgeting process £300k is anything but a guess, and (b) if the team know they have £300k they know they may as well spend it all.
But if our target was something like “increase our userbase for no more than 12p per user” then they’ve got much more scope for success. They may propose a large project or a small project, and in particular may experiment on a small scale to see which strategies are mostly likely to be successful before proposing anything. Either way it will be efficient. Where does the board get the figure of 12p per user? By thinking in ratios, and looking back to see historically how much new users have cost.
In the end these are the key points for me:
- A ratio measure provides a team or individual with more levers for success.
- Such measures can (and probably should) be specific to what’s important to the organisation.
- They can be used for relative targets, absolute targets, or limits.
- This really only becomes effective when the numbers are communicated throughout the organisation, so everyone knows what they’re working towards.
- Mixing financial and non-financial numbers (cost per this, or revenue per that) generates better judged and more efficient outcomes.
Innovation requires trust and freedom. But freedom only comes from trust, so the primary requirement for innovation is trust. And broadly speaking, the more trust you extend to a development team the more innovative they’re going to be. At the very least, innovation will not extend beyond the trust they’re given. Here are four levels of trust, and the innovation each can lead to.
- Trust = 0
- Trust with tasks
- Trust with milestones
- Interlude: Where we don’t want to innovate
- Trust to define projects
Let’s start at the very beginning, where trust = 0 and hence freedom = 0. “I don’t trust you to work by yourself; everything you do must be done in conjunction with me.” This is micro-management and generates nothing unexpected whatsoever. The only opportunity any team member has for creativity is during their bathroom break. We can do better.
Let’s extend trust a little bit: we allow our team members to complete individual tasks or deliverables that we specify — but let them determine how each task should be completed. This is work to a tightly-defined plan, and so any innovation will be under the hood — the internal architecture, the software design, and so on. But what you’ll see is what you expect. The tangible product, and every visible aspect of the product, is exactly what you imagined.
We can extend trust further still: we allow our team to deliver to clear product goals and trust them to work out the details of those product goals. This is where we start seeing innovation — as opposed to having innovation under the hood. Innovation of this kind is driven by project milestones or product principles, like “We want some personalisation features” or “The user must be able to share their work”. We want the development team — the programmers, user experience people, designers and so on — to be free to determine how to achieve this because it’s they who have the most intimate understanding of the product, how it fits together, what it’s capable of, and what it might be capable of.
In this scenario those project milestones or product principles come from outside the development team. It’s what they rely on product managers or other senior stakeholders for. Amazon does this with a press release: a description of the product in terms of end-user benefits, written to inspire and guide the development team.
If you’re thinking that this amount of freedom, from this amount of trust, is just hippy nonsense that leads to escalating budgets and timescales, then let me bring us down to earth.
One area where we almost certainly don’t want to be innovating is project management. There are plenty of good project management methodologies out there, one or two of which will be good enough for us, and we don’t need to be inventing any more. So we may not go in with a detailed task list and instead trust the team to define the tasks themselves; and we may not go in with any predefined product features and instead trust the team to work out for themselves what features will achieve the stated aims of the product; but to not go in with any idea of how we might responsibly spend our budget, manage risks, showcase progress, communicate with stakeholders, meet timescales, and adapt to feedback is downright irresponsible. And to generally deprive the team of project management expertise is foolish. Unless we explicitly want to invent a new project management methodology we really should ensure the team picks one of these tools and has the wherewithal to use it well.
Trust leads to innovation, but let’s be clear about where we want to innovate.
Now back to our regular programming.
We can extend our trust even further if we allow our team to define not just product principles or milestones, but entire products or projects. We can set an organisation-level goal such as “we want to double our revenues” or “we need to make our service a social experience” and then see how our team can achieve it.
A goal like this is important for two reasons.
First, it eliminates irrelevant ideas. Anyone in an organisation can have ideas, but we really want ideas that push our business forward. If my media company specifies that our goal is to make what we do more social then an interesting idea like making an iPhone game will be considered irrelevant because it doesn’t meet the goal of making our existing work more social. In other circumstances it might be promising, but our current circumstances are about being more social and that’s what we need to focus on.
Second, it allows us to compare alternate ideas. I might propose that people could be given virtual biscuits every time they share an article. Already this is more relevant than the iPhone game idea thanks to having to focus it on our organisation-level goal. But if someone else has a proposal to create common interest groups around our content then I’d say my virtual biscuits are looking pretty poor, relatively speaking. Our organisation-level goal has provided a scale of comparison.
And round again…
We’ve just extended trust to enable people to envisage an entirely new project. This is the very first stage of development — the concept. We also have to implement it. And this takes us right back to the beginning: how much trust do we want to give to the team? None? Just enough to complete a series of predefined tasks? Or do we trust them to devise the most appropriate means to meet the principles of the product? This is our chance to inject innovation again.
Lots of questions went unanswered in my last piece on putting ranges of projections into project proposals and I want to follow up on one of them here: What if our range of values isn’t something we’re happy with?
In my previous example our project had a revenue projection of £150k to £500k. But suppose the project was expected to cost £200k. Then that range encompasses both profit and loss. What can we do?
1. Collect more data
We may be very confident of our potential revenue, but perhaps we could be more confident. More research data should change our perception of what might occur, and so narrow the range of expectations. This might include customer research or research into other companies’ experiences.
Of course our findings may just serve to confuse the issue: we may collect six examples of other companies’ experiences only to find they differ wildly. In such cases we should be more discerning about how we see the data. For example it would be a good idea to learn more about each example and see which are closest to our own proposal.
2. Break down the problem
Our revenue projections (or other measurement) are likely to be a factor of several other elements. If we can break down the problem we can perhaps analyse those elements more easily. With more confidence over simpler measures we ought to be able to make tighter estimations of the whole.
For example, revenue may be seen as a factor of market size, typical customer spend, product visibility and ability of the product to meet customers’ needs. There may be market data readily available on market size and customer spend; we should be in a position to gauge product visibility if that’s a result of our own marketing; and focus groups or similar may be able to help us understand how the product is seen to meet customers’ needs.
3. Move the breakeven point
If we’re worried that we may generate £150k to £500k from a product or project that costs us £200k then maybe we should seek to reduce the costs. Of course if we reduce functionality or quality then there is likely to be some detrimental effect to the revenue. But we may be able to find high cost/low value items which change the game sufficiently.
4. Change the factors that influence the range
It may be that we can reduce uncertainty by actually influencing some of the critical elements. If it transpires that customers don’t see the product as meeting their needs sufficiently then we should change that. That may be by changing the product, or by improving our marketing.
Or we may find it’s useful to do the opposite of our previous suggestion of reducing cost: increasing quality (and adding a bit of cost) may pay extra dividends by increasing take-up. This kind of thing can be seen in places like the Apple App Store. I am at least twice as likely (probably four to five times more likely) to buy an app rated 4.5 stars than if it was 3.5 stars. Yet the difference in production cost to get to that level would have been far less than two-fold.
I’m sure there are more ways to respond to an insufficiently compelling range of possibilities in a project proposal. Overall, though, all these actions have the same beneficial effect I pointed to before: the relevant issues are surfaced so that they can be addressed appropriately and we increase the likelihood of our project’s success. And they are much more likely to be surfaced an addressed if we also surface that fact there is a range of outcomes, rather than project a single outcome.
Reading Doug Hubbard’s excellent How to Measure Anything I got thinking about the proposals I read and write, the problems they face, and how to fix them.
A typical proposal (I’m thinking mainly of internal projects) will make a claim like “This project will add 500,000 unique users in the first year” or “This will bring in £400,000 of new revenue by October” or “This will save £250,000 next financial year”. I know because I’ve written some of them. In choosing what number to use you don’t want to put in anything too low or the project won’t look worthwhile, so you tend to put in something that’s a little generous (but not implausibly so) and assume that all the contributing factors will work out for the best — the marketing, the timing, the user take-up, and so on. Depending on the audience you’re presenting it to you might even have some evidence for your projection.
The problem is, of course, that it won’t happen like that. Some things will work out well, some not so well. Everyone knows that, and if you’re facing a sceptical inquisitor then you’re just setting yourself up with less credibility before you start. Thinking back on the last such proposal I wrote I can imagine a conversation with a fantasy inquisitor that might have gone like this…
Fantasy inquisitor: You say this will bring in £400,000 before October.
Me: That’s right.
FI: Can you guarantee that? Can you guarantee £400,000 of new revenue?
Me: [fearing liability, and wondering which members of my family might get taken away if it doesn’t happen]: No, of course I can’t absolutely guarantee it.
FI: So what are you saying about the £400,000 if you can’t guarantee it?
What indeed? The truth is that will be the revenue if things generally go well. We might make even more. But realistically we might make less. Being honest with ourselves and our audience will help everyone. Let’s return to the scene…
FI: [repeating for the sake of the drama] So what are you saying about the £400,000 if you can’t guarantee it?
Me: I’m confident we’ll make that.
FI: How confident?
Me: Really quite confident.
FI: So you’re “really quite” confident it will make at least £400,000. And I suppose it could make us as much as… what? £800,000?
Me: Er, no I don’t think it will go that well. Maybe £500,000.
FI: So you’re confident it could make as much as £500,000. And you can be confident it won’t generate a penny less than £400,000, can you?
Me: Well, it certainly could make less, I suppose, if not everything went to plan.
FI: Would you be surprised if it didn’t make us anything at all?
Me: Yes, I would. Even being realistically pessimistic it really should make at least £150,000.
FI: Okay, so you’re very confident we’re looking at making £150,000 to £500,000 with this?
Me: Yes, I’m very confident of £150,000 to £500,000 of new revenue.
So our single (and therefore implausible-sounding) figure becomes much more realistic when we change it into a range, even if it goes down at one end. It also goes up at the other end.
I think this goes a long way in dealing with a sceptical audience becaues it is more realistic. For the same reason we should be more confident when we talk about it, too.
I’ll add two final notes. First, there’s an important question that needs to be asked, which I’ve omitted: What is your evidence for this? This may be calculations based on observations, historical data from similar projects in the past, market research, etc. Either way, it’s important to know that the figure(s) in question aren’t entirely random.
Second, the statisticians in the audience will point out that “very confident” is almost meaningless when we could say “90% confident” or attach some other actual number. I certainly appreciate that, and mostly concur. But for me a significant, easy, and meaningful step forward is to move from single figures to more realistic ranges. And this should promote more honest and confident dialogue.