My criteria were: low tech (well, as low as a spreadsheet can be); must be able to track historical changes; must not rely on complicated spreadsheet formulas; must be very, very simple. Manual copying is allowed. Remember: this is for simple projects.

Oh, and of course it must product a burn-up chart which looks a bit like this:

That’s a pretty straightforward chart for a spreadsheet. So to generate that we want a simple grid that looks like this where the scope changes from time to time and total done slowly grows:

I’ll call that the “chart grid”. The “Day” column is a sanity check only.

But obviously it’s going to start off different. It will look a bit like this, with only the initial scope known, and no work done (yet):

But where does that “Estimated project size” of 20 come from? Well, we also need to show our (initial) scope, and the tasks that need to be done:

The 20 is the sum of all the tasks. I’ve not included an automatically calculated “Total” row, but you could. I pasted the sum (20) manually into the “Estimated project size” column for the present day and a few days into the future.

Then when the first task is done we record it, including the date done (for reference), how much was done, and the running total:

We also need to copy the total into the chart grid:

Again, the 5 is copied manually, as I decided that clever referencing formulas were just more trouble than they were worth.

As time goes on we complete more tasks:

…and we record these manually in the chart grid, too. We read the table above as “On 4 Nov the total done goes up to 7″ and “On 7 Nov the total done goes up to 8″. So the chart grid then looks like this:

At this point the burn-up chart looks like this:

But then, on 10 November, we discover the scope needs to change. It’s important to me that we retain the information of the scope as it has been so far, so we do a few things. First, we change the title of the scope grid to reflect when it was valid from. We rename it to “From Fri 21 Oct 2011 to Thu 10 Nov 2011″:

Second, we move it down a few rows in our spreadsheet (or maybe move it to another tab) to make room for the revised scope.

Third we create the revised scope. This involves copying the work already done, changing the other task lines as appropriate, and naming it to show the date from which it is valid:

Fourth and finally we also record the new scope total (24) and work done so far on the chart grid:

The burn-up chart now looks like this:

And so it goes. Shortly the scope changes again. Again we revise the title of our scope table, move it down out of the way, copy progress so far into a new scope table, and show the rest of the revised scope in this new table, adding progress as we go:

We also record the revised project size estimate (29) and we record progress in the chart grid:

And our burn-up chart looks like this:

If you want access to the formulas (there aren’t many) then this completed example spreadsheet is available as a public Google Doc.

But then he asked in a different way: what were my experiences as a project manager before and after implementing Agile? What differences might he expect? That was a great way to look at it. Here’s a little of what I said…

Less waste

Before Agile I would find myself running projects with wasted activity. We’d spend the first few days creating the database schema and libraries. And then by the time we got to the end some features would have been dropped — maybe because we had to go live sooner than we’d have liked, maybe because requirements were explicitly descoped. And looking back we’d realise we spent days implemented the underlying functionality for those unused featured. Similarly I would have specified interfaces to high level of detail, and yet those interfaces would have changed during implementation. Again, wasted time up front.

After Agile, that doesn’t happen. If a feature is needed it’s implemented top to bottom in one task. If it’s not needed none of it is implemented.

Better software

Before Agile, if that library or database schema was still in use, then often it was in use poorly. We’d implemented it based on expected requirements, but when the day came those requirements turned out to be slightly different. Maybe because of a lack of understanding, maybe due to an actual change. But there wasn’t time to change the earlier work, and we could still manage with a workaround, so that’s what we did. And the result was software that worked, but included serious compromises, to the extent that it was really difficult to evolve beyond the original launch.

After Agile that doesn’t happen. Well, not so much. I’m not sure I’ve ever delivered perfect software, but I’ve certainly — consistently — delivered software that can continue to grow with the requirements of the client and their business.

Fewer difficult change conversations

Before Agile, conversations around mid-project changes were very difficult. If the client wanted a change then they they would hear me act like an unhelpful car mechanic. I’d suck my teeth, exhale, and say “We-e-e-e-ll, it’s gonna cost you….”. We’d worked so far based on a different assumption. So, yes, they could have what they wanted, but at significant cost. Or they could have a compromised version at a reduced cost. What was it to be? Significant cost or compromise? Not a great decision.

After Agile, change is the norm. If you want a change (which might be a new feature, since that’s a change to the software we’ve delivered to date) then we’ll prioritise it appropriately. If it’s going to take 8 days and you don’t want to increase your budget then just take out something else, that’s less important, that’s also 8 days. All features should be interchangable, so you can change your mind a very short notice.

It’s not a perfect world

I was at pains to add that I’ve still had difficult projects before and after Agile. Agile isn’t perfect, and neither am I. (“And,” said my friend, “you’re still dealing with people.”) But undoubtedly the chances of success are much greater, and the general level of satisfaction is much higher.

My objection is two-fold. First, agile in any organisation needs to be explained and communicated well, such that it can be understood and embraced by people of all skill levels. If this is not done then that’s either a failure of those people who are doing the communicating, or it’s an sign that the delivery approach used in that organisation is ad hoc and/or weak. That’s a failing within the organisation.

But (second) if, despite the very best methodological structure, and despite the finest communication, agile still cannot be used by less experienced or less capable people, then it means agile is not the delivery solution that our industry needs. Indeed, it means that it’s useless for 50% of all software teams (given that 50% of any population is less than average, by definition). This would be a failure of agile.

Generally, the claim “you need to be better” is a cop-out for any methodology: “My technique for delivering software better involves hitting the keyboard with a large hammer… what do mean it doesn’t work for you? You’re obviously not good enough…”

However, when I hear people say you need to be more skilled to do agile I really think it’s a momentary failure on their part. Sometimes there will be an element of snobbery there. Sometimes it’s failing to recognise that weaker teams can produce results that are better relative their own norms even if they aren’t as effective as the stronger teams — strong teams will always produce better results, but weaker teams are able to improve.

We also need to acknowledge that “better” is not measured on a single dimension. Agile places a greater-than-usual emphasis on human interaction — someone who is stronger at this will do better in an agile environment. In fact, someone with average technical skills plus very strong interpersonal skills will be more desirable for an agile team than someone with very strong technical skills but zero interpersonal skills.

The claim that agile might be best restricted to “better” people was address by Martin Fowler seven years ago. In that piece he offered an open verdict, which was shaped by the context: agile was at the time newer, and less proven.

Today the context is different, and agile is more mature. It has evolved with a huge number of supporting practices, variants, tools and advocates. With that vast body of collective experience I think it is now much more able to spread and be embraced by increasing numbers of practitioners… whatever their experience.

This is not to say seamless delivery into production itself is a neglected subject — it certainly is not — but it’s often regarded as a topic distinct from the agile software development process, and often not implemented. It’s easy to see why this is: in part it’s because it’s quite possible to demonstrate working software without it being in production. In part I suspect it’s also because the last mile is typically the realm of the systems integrators and administrators, who typically sit (or are sat) outside of the software process. And, let’s be honest: getting something into production is difficult, getting it into production regularly is very difficult, and so it’s good be able to claim the agile badge without having to sort that out.

But getting something into production again and again and again is a huge benefit to the core goals of agile: until something is actually being used for real by its end users it’s very difficult to make further calls about how it should evolve. By doing this we really get to understand what counts as valuable. And those end users are rarely the business analysts or internal customer that a lot of teams deal with.

Unfortunately a consequence of pushing for a seamless last mile (or continuous delivery, or however you would label your target) is that it forces you to be clear about what success really is. For many software teams a demo in front of the internal customer is success, even though that’s often a cop-out. Getting the thing in front of your end users is what should be classed as success, but that’s often difficult to deal with, because it means having to respond to potentially unwanted feedback in the middle of your project. And yet if the agile software development is about delivering value — regularly — then the value is really only measurable when it’s in your real end users’ hands.

“It’s a mix of ignoring them [non-Agile managers] & making shit up” #ProductTank How is this consistent w/ Agile values of openness & trust?

There’s a conference-worth of material to unpack from that question. You can make up your own responses, whether cynical, pragmatic, witty or all three. I wasn’t at Product Tank, so I can’t take issue with that in detail (though I would caution readers: I know Tom, and his presentation will have been as much about entertainment as education and information, so take that statement with some amount of salt).

However, regardless of its literal truth about alphagov specifically, there is a truth in there about projects in general: if the idea of agile is used as a trojan horse to allow bad behaviours, then any success will be shortlived. That thing masquerading as agile won’t scale to other teams; it certainly won’t scale to larger projects. Agile might be seen as the success factor in the initial project, but the inevitable failure of future projects will ensure its adoption is halted pretty quickly.

This is true for small and large organisations of all kinds, and is one thing that James Yoxall touched on in his presentation to the 2011 Agile Business Conference.

When I manage development teams I see one of my responsibilities as being the angel on the shoulder of those developers, urging them to “do it right” in the face of the project manager who’s urging them to “do it now”. But project managers can be angels, too. Some will be interested only in the success of their own project. And some will want to ensure that good practices are recognised, learned and adopted by others. Unfortunately those two views are sometimes at odds.

Specs are useful for filling a communication gap. That gap may be one of team distance (the BA to the vast army of developers, too many to speak to all at once), or time distance (we’ll write this down now, because we may forget it in 3 months’ time when we start the work), or organisational distance (we’re going to give this to you, third party company, because this is precisely what we want you to build for us). There will be other examples.

But if those gaps don’t exist then you probably don’t need a spec. In particular, if you have four people in the company, and you don’t yet have any software, and are still validating your product idea, then a spec isn’t going to be useful. There, the best thing is to write the smallest thing you can, try it out, and repeat.

None of which is to say that all documents are bad. Far from it. It’s a good idea to write down a strategy (lest you forget it in the heat of the moment), an architecture (which you should aim for but may want to adjust), and some principles of various kinds (UX, engineering, product, etc).

But a spec is too often a document we could do without.

These notes are terse, but in true agile fashion I’d suggest there is little additional value in turning them into beautiful sentences…

Delivering Agile in Government: Learning lessons from the commercial sector

Jerrett Myers of the Institute for Government:

• The IfG considers itself less of a think tank, and more of a do tank.
• Classic government problem: benefits are accrued by one organisation within government, but the costs are felt by another.
• The process of IT in government is fundamentally broken: Complex problems with numerous dependencies; Technology and priorities are rapidly evolving; Government processes are slow and unresponsive; …which all added up means requirements are obsolete, high cost of change, over-customisation, low user focus.
• Challenges include…
  • Cultural ones: Taking responsibility, and not being afraid to make decisions quickly; Dealing with high level outcomes rather than clearly defined requirements.
  • And skills shortages: No government CIO approached could suggest a single developer within their sphere to work on the pilot project.
  • And governance: The typical approach is to require 100% agreement before any work can begin.

James Yoxall, of IndigoBlue:

• It’s common for agile adoption to happen only at the fringes of the organisatoin, and if it’s not tackling problems at the heart of the organisation it will always stay that way.
• Projects are commonly implemented in agile to avoid normal governance.
• Good to think about agile as incremental delivery: an incremental flow of deliverables (as opposed to the traditional approach of a single deliverable). You don’t need a wholly effective “team in a room” to be successful at this, because you’re simplifying the problem. Simplifying the process means descoping until your first deliverable is a minimal system, and then you add the additional scope incrementally.
• This requires agile governance, which is a new skill. It requires talking about and explaining the individual steps of the journey, not just the end goal. It also includes showing the senior stakeholders that the many incremental steps add up to something adequate and that meets the overall goal.
• Agile is also: uncertainty management. Preparation is important, but you don’t necessarily need to do all your preparation before all your doing.
• At the start of project be clear about what your uncertainties are, and what their impact is, and decide if you should hold back on the doing before you have cleared up those uncertainties. Sometimes analysis resolves these, sometimes early actions and feedback resolves them. NB: Uncertainty is not the same as risk.

Agile Governance

Chris and Andrew are from Axa and nlighten respectively, and they reported on their experience of introducing appropriate governance structures for agile projects into Axa. I originally recorded these tweets:

• Governance for: Strategic (vision) alignment, risk mgt, resource mgt, performance mgt, value delivery
• Performance governance = delivery-centric measures, retrospectives, continuous improvement
• IT governance is the demonstration of control [is that the same as the actuality of control?!]
• Understanding != documentation, discipline != formality, quality != bureaucracy

My major take-away point, though, is that it’s important to understand what governance is for, and build your control structures accordingly. And one helpful way to determine what it’s for is to think through the consequences of there being no governance structures.

You want more than that? Well, you had to be there.

Calling for an update

Steve’s starting point is this:

In 2011, “working software” is not enough. Unless the customer is delighted by the working software, the future of the business is not bright. Customer delight has become the new bottom line of business. Firms that succeed in delighting their customers like Amazon [AMZN], Apple [AAPL] and Salesforce.com [CRM] enjoy happy customers, soaring profits and workers who can see meaning in their work. Firms that don’t experience the opposite.

His solution is then that “customer delight” should be embedded into the agile process — and in particular into the Agile Manifesto. It’s pretty hard to disagree with many of Steve’s broader points: that goals need to be embedded deeply in an organisation to last, that product development is only really “done” when the customer’s happy, that customer happiness can be measured, and more.

But is the Agile Manifesto the right place to embed the goal of customer delight? For one thing, his exemplar companies demonstrate nothing about agile methodology. Amazon, Apple and Salesforce.com delight customers with their customer service, their hardware, and their ubiquity respectively. They may use agile development techniques, but their USP and their market cap (Steve’s indicator of excellence) is not an extension of the Agile Manifesto. Similarly you’ve got to have a pretty peculiar view of the world to think that Walmart, Cisco and GE (Steve’s examples of companies which aren’t performing) would shift to a stellar performance if only the Agile Manifesto could be updated.

Antagonising developers

Because Steve seeks to rewrite specific agile practices, Kevin Ross and Nathan Gloyn get stuck into those details and show why he’s mistaken. And yet aside from the specifics there is a common underlying feeling expressed by both Kevin and Nathan, which is that developers have a hard enough time as it is, thank you very much, and they can only do their job effectively by trusting that their colleagues (the product owner, the company board, etc) are also doing theirs effectively.

To that end, here’s Nathan:

I have multiple customers to satisfy ranging from immediate and upper management through external clients to actual users of the software/website who may (or may not) be customers of the business [...] The PO is ultimately responsible for prioritizing/ordering the work to be done by the team and deciding what is of the most value to the business in helping delight customers but it’s a collaborative process that should involve the whole team and where appropriate also include the customer, but somebody does have to make a decision and that responsibility lies with the PO or PO team

And here’s Kevin:

If the customer defines the user story, as this is a significant requirement to deliver accurate software using most agile incarnations, then it should be a given that the customer should be delighted with the result. If not, this is a symptom, and the cause should be addressed. [...] How can a developer know what delights the customer if they did not define it as a story or acceptance test? [...] We rely on a product owner, and they make decisions, define stories, and define priorities. If your “product owner” can’t do that, you have to address it directly not try to work around it. We all have a role, and you have to depend on them to fulfill their role.

Reading the debate further it’s clear that all parties agree that delighted customers is the ideal end goal, so why the strong disagreement? By poking at the Agile Manifesto Steve has brought out the worst in Kevin and Nathan and it would seem, most of the audience of Agile Alliance 2011. Here is a bunch of educated, engaged developers who seem to be saying “Delighted customers? More than my job’s worth, mate.” Clearly this discussion is going awry.

Software development is only part of the chain

The source of the problem is that software development is only one link in the product chain: The pipeline of work coming into the development team has to run smoothly; the items going into the pipeline have to be the right things (e.g. things that will delight the customer); the end product has to be marketed well (would Apple have such legions of fans without great marketing?); customer feedback has to be balanced with product or company vision; and so we go round again.

That one link — software development — is so difficult, and has such a history of problems, that the practitioners had to produce a manifesto to point the way forward and keep them on track. When you start poking at the island of salvation that is the Agile Manifesto the practioners are going to get jumpy. The manifesto is of specific interest to software developers, so trying to add in “customer delight” looks like it is giving that responsibility to the developers while ignoring product managers, CEOs and marketing execs who feature more heavily in the other links in the chain. No wonder the developers are going to say “Hey, what about those other guys?”

No doubt we should all raise our game when it comes to product delivery. This might be achieved by dismantling that chain and bringing everyone together to think more effectively about the customer. Or it might be achieved by encouraging all team members to continually check and verify the current thinking, just to make sure everyone is focused. But it is not achieved by taking a philosophy of software development and turning it into a methodology for successful business.

Software developers need principles for effective software development. They may also utilise principles for creating a great business though great products, and perhaps those principles could be produced by adapting the Agile Manifesto. If so, such product development principles should be meaningful to everyone in the product chain. But principles for software development would still be necessary, and they would be different from principles of product development. They would be applied at different moments, for different purposes. They shouldn’t be confused.

• The business decided that the team should be “coding” for a full 7 1/2 hours a day and continually checked that this was the case.
• The management take care of the estimates
• The developers are not part of the planning process
• Estimates are made on a best case scenario
• [Management] think when we are talking about architecture and code quality that we are not working hard enough.
• They do not understand that if time is not given to extensibility and maintainability, that delivering software will become very costly

There are other things, too, but those are the ones I want to highlight. Because the correct response to the question “Is it agile?” is “Wrong question”. It’s a bit like realising the aeroplane you’re in is heading towards a crashing landing, and frantically checking your ticket to see if you really did book a window seat after all. Never mind about whether it’s agile — this is just not even sensible.

All the practices above are the kinds of things enforced by people who aren’t very experienced with running software projects — although I would agree some of them seem like common sense to an outsider, and all of them are the kinds of things software managers can get pressured into by stronger outside forces. But they’re a long way from what should be happening in any well-run development environment. Let’s address them in order:

• Yes, coding is absolutely essential. But so is co-ordination, planning, revising, sharing knowledge, flagging complexities, helping colleagues and a lot of other things that don’t involve typing.
• If the person doing the job doesn’t make the estimates then there’s no reason to expect the estimates to be right.
• If the people doing the job aren’t involved in the planning then there’s no reason to expect the plan to be realistic.
• Estimates aren’t predictions.
• If you don’t design the system right, and keep checking and refining that design, then everything will be more difficult for everyone. Getting this right is called “architecture”. Poor quality code also usually makes things more difficult for a lot of people, so it’s worth spending time on.
• It is very easy to cut corners early so as to increase development costs later.

The second, third and fourth points here should be common sense, while the rest probably aren’t obvious to a non-software person. But in the world of software development getting these right is essential — it’s just good practice. And in doing so you would make the life of a developer like Paul (and the projects he’s working on) a whole lot better. If you want to be even more effective you could adopt agile practices, too. But you can still be sensible without being agile.

1. Wonky estimation

At the Guardian we’ve tended to fret when individual story cards start running over their estimated times. During the R2 project we even instituted a rule that any story going over 25% of its estimate got escalated.

By contrast Kelly said his experience was that teams would routinely misestimate stories, while consistently making their target velocity and meeting their delivery promises. In other words, there was consistency in poor estimation, but also underestimates would consistently balance out overestimates.

It seems contradictory that failure in the details should allow for success in the outcome, but that’s what happens. Kelly put this down to the magic of Fibonacci numbers.

2. Ignoring estimation by role

Meanwhile, we have generally broken estimates down by role: a card might be estimated as 3 Java days, 1 client-side day, and 2 testing days. This follows from iterations being similarly described as having a certain number Java days, client-side days, and testing days.

Kelly’s approach does away with that: the team is a whole team, estimates are one number, and this follows from the iteration having a known capacity which is also one number. The emphasis is on team members acting together and helping each other out, and he didn’t seem particularly vexed by the idea of one team member — a back-end developer, say — running out of work in an iteration because the workload that fortnight was particular front-end-heavy.

The approach I’d used in the past suddenly had echoes of old waterfall-style project management: trying to squeeze the most out of each individual by trying to fit everyone’s time together like a jigsaw.

As an aside, I’d say the “different roles, different estimates” approach has the advantage that it allows individuals to stick to what they enjoy and ensures quality work continues in their areas — client-side developers enjoy doing client-side development, and no-one could expect a back-end developer to as good a job there. But equally, it avoids the issue of trying to break down barriers between roles. And I suspect that one thing that helps break down those barriers is ensuring the teams remain as consistent as possible. But that’s a topic for another day.

The common theme

The common theme to both these differences — not fretting over individual estimates, and having a single estimate for each story — is that the intended outcome (end of iteration delivery) is successful even though the details are fuzzy or downright wrong. This is counterintuitive, but I also find it liberating. It’s always good to find new approaches to a topic you think you know well.