Category: Agile

An ABC of R2: I is for iterations

…which lasted two weeks and culminated in a full new release of our software. I’ve written elsewhere about what happened in one week of a particular iteration in June 2008.

However, our R2 iterations didn’t just involve implementing software. At the same time each team was also working with a business analyst and end-users to plan and clarify the work for the next iteration, and sometimes getting the last iteration’s work through its systems tests prior to release. As one of the team leads said to me, “we work in three timezones”.

At the end of an iteration the release went out. But our releases didn’t always reveal much that was different — either because we were waiting to reveal it as part of a forthcoming launch, or because it was part of our internally-facing systems. That can be an odd feeling for the development teams. In previous lives I’ve worked on projects where the launch has the whole team working right up to the last moment and so also signals the time when you can collapse with exhaustion. But our interleaving of “timezones”, and releases well ahead of launches, changed that. The launch of, say, the Sports section was a big moment for so many in the company, but most of the people on the development team had finished the work some time before and were spending more time thinking about the launch that followed.

An ABC of R2: C is for changing requirements

…which are a fact of life — certainly if your life revolves around developing software. During R2 there was a 40% churn on requirements. That means by the end of the project 40% of the work we had done had not appeared in our initial plan — some things were dumped, new things were introduced, and much was significantly altered.

The Agile way is to embrace change, and more than that it’s to structure your working practices around enabling and encouraging it. There are several ways we make this happen. One is to make sure that every task has tangible value to the end-user. Another is to prioritise the tasks so that the highest value work is delivered first. The second follows from the first: only if the tasks are individually valuable can they be arranged and rearranged.

One example of handling changing requirements is how we introduced keywords. We always believed these were important to our work, so we developed them very early — other elements of our plan were less certain and so might change, but we could safely defer such decisions until closer to the time we’d have to build them.

Managing keywords would require some slick tools if they were to be rolled out to more than a couple of staff. But our first cut was very primitive: it involved someone typing data into an Excel spreadsheet and then uploading that to a web page which would digest the spreadsheet and either process the data or report errors. It was not user-friendly. But this approach did have many virtues:

  • It was relatively quick to develop;
  • It was sufficient to enable just one person to manage the keywords, which was all that was needed then;
  • It allowed us to build further things based on keywords, such as a little component next to an article which listed the keywords associated with it;
  • It showed up real, unexpected problems arising from real-world use, that we would not have foreseen during planning;
  • It showed us what really was and wasn’t going to be important when we built a slick tool for wider use.

A 40% churn on requirements doesn’t mean there was a 40% budget overrun or time overrun, by the way. We came in on time and on budget. That’s another consequence of having working practices that embrace change rather than merely tolerate it.

An ABC of R2: B is for business analysis

…which means different things to different people. In our case it meant extracting requirements and turning them into something that could be implemented.

Business analysis is often misunderstood when it’s used in an Agile context. Agile people often think it’s not necessary — after all, they say, the analysis is best performed by the developers working closely with the customer as they go along. Many non-Agile people often think the analysis should be done up-front, written down, and then the workload on the business analysts (BAs) eases off as everyone else goes off to implement what they’ve written.

In fact business analysis was one of the most demanding jobs on R2. Analysis was indeed done up-front, at the start of the project, but only to a level sufficient to estimate the work, not to a level sufficient to implement it. Ahead of each fortnightly iteration the BAs would specify the forthcoming fortnight’s tasks in much greater detail, sufficient to be implemented. At the same time they would be dealing with queries about the iteration currently in flight. Thus at any one time they had to deal with work being planned and work being implemented.

In both cases they would be working with a very diverse group of people. For work being detailed the business analyst would take requirements from our business users and present them to the technologists for estimation; for work being implemented they would often take queries from the technologists and present them to the business users for guidance.

You could say the process of business analysis was often like being piggy in the middle, yet the personal skills of the individual business analysts (technical insight, strategic understanding, and not a little diplomacy) was key to the success of the project. The cohesiveness of guardian.co.uk — both from the outside and the inside — is in no small part due to the talents of our business analysts.

What management buy-in really means

We work using agile development processes, which obviously needs the buy-in of internal users and project sponsors. But this jumped out at me when I read it. It’s from Carolyn McCall, Chief Executive of Guardian Media Group, which owns the company I work for. She was announcing a £15m investment in our digital business, and it was reported thus:

“What we’ve done so far is our own version of web 1.0, but we want to continue to web 2.0 and what comes after that,” Ms McCall added. “We need to be agile and ready to change.”

I suspect that phrase, “We need to be agile and ready to change”, is not a coincidence. You can trace a strong lineage between the agile development work going on in Guardian Unlimited and that little sentence in a speech to the OPA. Let me tell you how I think it got there.

A story of a sentence

A long time ago we began using agile processes in the GU tech team. It was a new way of working, but it seemed to address a lot of our problems — actually, the usual ones you get in most software organisations, such as knowledge silos, lack of flexibility, impossible deadlines, and so on. Changing what you know is always a tough choice, but the GU management team were very supportive of the move to agile.

In particular, our boss at GU at the time, Simon Waldman, was very interested in its implementation and evolution. (Simon’s since moved to GMG.) Over the months and years he enquired, queried, provoked, but was never less than encouraging and constructive. (At one memorable annual review to the GU staff Simon devoted an entire slide to explain how we had stealthily removed a problematic database table. Our DBA was thrilled; I have no idea what the attendant sales people and journalists made of it.)

And agile clearly worked for GU. One notable change was that planning meetings took on much more of a business focus. Far less about technical dependencies, much more on what functionality we wanted to release when.

And then we began we work on our new Travel site (released in November). It featured a new design and new commercial opportunties. More people around the company needed to scrutinise this as it started up, because it was a pretty big project and we had a much greater responsibility to be rigorous and transparent in our delivery. We saw agile methods as being critical to the work’s success. It would allow us to delay decisions to the latest possible moment and therefore produce something that was much more relevant — both editorially and commercially — to our audience and clients.

I therefore spent some time explaining agile to various company directors… but not as much as you might expect, because clearly the principles and practices had been discussed and understood well before. I ended up having the kind of conversations about agile development with non-technical senior managers that I wish I could have had with more technical job candidates. Word was getting around. The business people scrutinising the work on the Travel site understood the process’s business benefits, and they understood how the low-level practices would provide those, and they knew those practices were working successfully within GU at the time. And indeed, the project was blessed by the the company directors, including Simon and our MD at the time… Carolyn McCall.

You can see how that word “agile” has been thrown around a lot inside the company, and was — at least then — strongly associated with Guardian Unlimited and therefore its innovation (which I’d like to think is almost synonymous with “daily work” — ahem), and it’s stuck.

Now I can’t claim that Carolyn knows what, say, test-driven development really is. I suspect she has higher level things to think about. But if it came up, I think she’d understand our practices such as refactoring (making lots of tiny internal restructurings to produce a smoother, more manageable system), continuous integration (ensuring our changes are constantly integrated into the day-to-day work, not siloed) and retrospectives (frequent reviews and actions to improve). And, come to think of it, TDD, which is really about supporting and enabling change.

It seems Carolyn understands “agile” in a way that I wish more technical people would. It’s about change, and about supporting change (“We need to be agile and ready to change”).

The hard work starts now

And what of the future? Well, agile development helps us deliver responsibly to the business — and if someone invests £15m in you, then you really need to deliver responsibily. Again, that kind of investment only happens if there’s confidence in your ability to deliver. Can we offer that confidence? Well, just this week I was in a meeting with Tom Turcan, who is our Head of Digital Media Development. The conversation went something like this. Note that development for this particular thing is already about 30% of the way through:

Expert user: “…So in summary we’ve decided that features A, B and C aren’t that important after all, and we’ve replaced them with features J and K.”

Tom: “It looks like A, B, C total 7 ideal days, and J and K total 8.5 ideal days. So that’s a net increase of 1.5 ideal days. Is that right?”

Expert user: “Er, yes.”

Tom: “So we can’t really allow that to happen. We can make changes to make the work less, or keep it at the same level, but we mustn’t start increasing it. Can you find 1.5 days’ worth of features to remove?”

Expert user: “Yes, I should think so.”

Tom: “Okay, we’ll make a note to see what you’ve decided next week.”

We couldn’t have had that conversation in a more waterfall environment. By the time we’d have reached this point the groundwork for A, B and C — the database tables, the DAO layer, etc — would have been done, and it would have been wasted effort. Instead our expert user is able to make decisions later, and Tom is able keep to keep the work under control, giving the rest of the business the confidence it needs. Our agile processes enable that to happen. And all that is known by our developers, and by our managers, and that same confidence is shared by our group Chief Executive.

Postscript

By the way, there are other things going on this week in and around Guardian Unlimited. You might want to take a look at Jeff Jarvis’s commentary on Alan Rusbridger‘s commitment to digital and his interest in being flexible. Also, my esteemed colleague Neil McIntosh tells how GU saved a cat, and offers a word of advice to a researcher at News International.

As a friend said to me the other day, “I know when you’re busy at work — your blog goes quiet.” So back to work now.

Is it agile? Check the fixed points

One of the problems with agile development is that it’s subject to changes, so means you’re in danger of changing it into something which isn’t agile. How do you know when you’ve gone too far?

This isn’t a problem for seasoned agile practitioners, but it’s a concern if you’re just starting out, or working with people who are. It’s certainly something I found looking back at my early agile days. There are some non-obvious or difficult agile practices which one might omit, but which make a big difference. How do we know what to watch out for?

My suggestion is to ensure there are some fixed points. Then you can evolve the process without straying from the ideals. My rule of thumb for creating fixed points is this:

Could I stand up at an international conference and be sure that my process would be generally indistinguishable from that of any of the speakers?

With that in mind, here’s what I’d want to be sure I did before I stood next to my peers…

A fixed point, by lobo235

Daily stand-up meetings at a set time

There are three things here: daily, stand-up, and set time. Making the meeting daily can be difficult, but having spoken to people who hold weekly meetings it’s pretty clear that they take up the same time in the end. In addition, daily communication keeps people veering too far off the track.

Insisting the meetings are stand-ups might be difficult and there’s no clearly tangible gain — after all, if you keep the meeting to 5 minutes, who cares whether you’re sitting or standing, or bouncing on one leg? But this important to me because it instills a sense of urgency. A sit-down meeting fails to distinguish us from our document-heavy, procedure-driven colleagues. It’s entirely psychological, and all the better for it.

Making the meeting at a set time can be difficult, particularly if the lead has other responibilities. But this is important to me because it forces people to focus on the project. It says “The train’s going at this time — be there or get left behind”. It tells people the project won’t wait for them. It keeps the team to a rhythm.

Iteration kick-offs and/or close-downs

It’s surprisingly easy to get away without these: some of last iteration’s story cards might not be finished, so the team just carries on; new story cards become available when the old ones are done, which could be any time. Iterations blur.

But a formal meeting to mark the iteration boundary again ensures the project and the team keeps to a rhythm. Coming up to the boundary it ensures people know ahead of time that they’ve got something to aim towards, and deadlines always focus the mind. Entering the new iteration it gives people the chance to think afresh and renew their goals.

Mid-project retrospectives with the whole team

An end-of-project retrospective is good, but mid-project retrospectives are agile. They ensure the team really is seeking to improve not just for the next project but for the current one. It can be difficult to see the benefit of retrospectives (another time-consuming meeting), nor do I think they’re not very obvious. I certainly don’t recall them cropping up in the Extreme Programming Explained or in Agile Software Development with Scrum. One way to keep retrospectives effective is to ensure any actions are (a) objective and (b) assigned to named individuals. This way improvements will be much more tangible.

Continuous integration and testing

This is certainly an obvious feature of agile, but it’s also difficult. If the project is anything more than a few days long it’s worth spending the time up front to get the continuous integration and testing working. This ensures rapid change and frequent delivery. Otherwise it’s very likely that features and fixes will regress, that this won’t be discovered for long periods, and fixing the regression will take a long time, too. Agile is about frequent delivery and constantly moving forward at a regular pace. That can’t be done without continuous integration and testing.

Frequent customer demos

Again, not necessarily obvious because while agile talks a lot about on-site customers it doesn’t often talk about the common case of customers not being around as much as you’d like. Frequent mid-project demos are the next best thing — at Guardian Unlimited we call them “show and tells”. A customer demo has two benefits. First, it steers the team towards feature-driven development rather than architecture-driven development. (It’s very hard to impress a customer with a data object layer.) Second it encourages prioritisation of the next batch of features.

Anything else…?

There’s much more you need to do to be sure you’re being agile, but to my mind the things above are easily missed. So next time I create a team for an agile project I’m going to make sure we start with some fixed points, and evolve round those.