I’m leaving the Guardian

I’m leaving the Guardian. It’s a sad time for me, but also one of great pride because I’ve got a chance to look back on eight years of digital growth and recall some of the wonderful things we’ve done…

The way we were - Photo by Kaptain KoboldEarly days

I introduced agile development to the Guardian team in 2003. I was not particularly an advocate of agile before then, but I saw it addressed many of our needs — the need to continually evolve our technology, to spread knowledge within the team, to deliver more predictably, and more.

But I couldn’t have done that without the trust and support of Liz Sarginson and Stephen Dunn, who were the ones who brought me into the organisation. I pitched it to them, and they could have said no. Or they could have said “Yes, but”. But they didn’t — they gave me their trust and support, and we made it work, although it was difficult — particularly for the developers, who need to deal with then-challenging ideas such as test-driven development with legacy code, and pair programming.

The trust they extended to me then has stayed with me since, and it’s something I try to remember when people come to me with their ideas now.

Slaying dragons…

It was the development team that identified our greatest challenges and convinced me that we — or, rather, they — could take them on. Those challenges were eliminating our most resource-draining legacy systems.

The first was updating our CMS tools to work in a more modern browser. At the time the tools were tied to Netscape 4, which came out in 1997. The project turned out to be much bigger than we’d guessed, because our own tools weren’t documented and we kept finding obscure features which seemed essential for someone, somewhere. But we managed it without compromising our ability to deliver shorter-term, more immediate things along the way.

The second was eliminating a database table. The table itself was originally introduced as a simple performance optimisation, but it had grown unexpectedly over the years, and it fired so many triggers that it was now making some of our tools so slow as to render them useless. But references to the table were scattered all over the vast codebase. It was the work of months to eliminate all references to it and finally to drop it. But we did it, without impacting any other work, and we popped a champagne bottle open when our DBA successfully typed DROP TABLE… and the prompt returned without complaint.

The third was the elimination of our legacy CMS engine, which ran our site’s thousands of templates and scripts. Over the years it had been configured and customised far beyond the vendor’s expectations and no longer fitted their roadmap. Also, it denied us reliable release mechanisms and modern version control for the code we put in it. We replaced it with an open source alternative which still ran the same templates and scripts. The migration was gradual, steady, and every step had a rollback plan.

…and building castles

After slaying those dragons of the past we were able to build for the future. We took on our biggest ever project, dubbed R2, to rebuild and redesign It introduced much more flexible templates and content management for the editorial teams, more flexible advertising options for our commercial teams, and more modern technologies and processes for the technical team. It lasted two and a half years, delivered new code into production reliably every fortnight, peaked at over 100 people, and came in on time and under budget.

It was also designed and built with technical integrity: our mantra was to be “of the web, not just on the web”. In the many months that followed we released a full-content API, a plugin framework, a new commenting system, and innumerable other products and features, large and small.

It’s the people, stupid

But beyond all the things we did, it’s the people I will remember with greatest pride. It was the team that stepped up to the mark and slayed the legacy dragons; it was people across the organistion who made our major rebuild and redesign a thing for the future; and it is trust and strong relationships within and without the technology team that allow ideas to continually flourish.

All long-standing media organisations face an uncertain future. If any of those can see it through then it’s the Guardian, primarily because of its incredibly talented people.

Agile Awards 2011 shortlist

Agile Awards 2011: ShortlistedI returned from last week’s holiday to an e-mail telling me that the Guardian has been shortlisted in the 2011 Agile Awards. In not one, but two categories: Best Agile Team, and Best Use of Agile in the Private Sector. I’m thrilled, because it acknowledges that success really is a team effort, and while you’re in the middle of the maelstrom it’s difficult to see if you’re making any kind of difference. Clearly the Guardian team has been recognised.

I should also blushingly acknowledge that I’ve been shortlisted for Agile Project or Programme Manager of the Year. “Blushingly” because the crowd I’m among is great indeed. Congratulations to all those people, organisations, and teams.

The awards dinner is on Wednesday 5 October, in London, and if it’s anything like last year’s event it will be a terrific night. And if that wasn’t enough, it also benefits two charities.

Guardian Hack Day 2

There isn’t too much I can add to Kevin Anderson‘s extensive coverage of Guardian Hack Day 2 [1, 2, 3, 4, 5, 6]. But here are a couple of things…

First, huge congratulations to Matt McAlister and the whole organising team. Hack Day 1 set the bar pretty high, but the team managed to meet that, if not surpass it.

Second, a terrific change from the last one was that so many people seemed to be sharing skills and ideas  across organisations. People from, the BBC, Inkling, Google, Talis, Omniture, Rabbit, Guardian and elsewhere all mixed it up to produce some really innovative stuff. Added to this, there was a healthy number of people who don’t claim to be programmers creating some very interesting hacks, thanks in part to Simon Willison’s list of Hack Day tools for non-developers.

So among other things we got a tracker of the most controversial articles, a page of search terms that people are using to find — live — a social network for dead people, and much, much more.

The photo here shows Matt under the gaze of the Guardian’s inspirational editor CP Scott — or rather a bust of him. I’ve put this photo and others in a Flickr photo pool. If you were there, please feel free to contribute.

The Open Platform: It’s thanks to individuals

The Open Platform trends on TwitterYesterday we launched the Guardian Open Platform, incorporating the Content API and the Data Store. A lot’s been said already about this — and more will be said — so I will add only a few words of acknowledgement. Many have pointed out (Tom Watson, Jeff Jarvis, ReadWriteWeb, Fast Company and others) that it’s a serious move by the company itself. So I really want to spend a moment to reflect on what that really means.

It can really be best captured by the questions asked by the crowd at the launch event: What if other news organisations want to reuse your journalism? Will the ad network really work? Do the numbers add up? These and other questions were asked at the time and were asked inside the organisation in the months leading up to the launch. The questions span journalism, advertising, finance, technology, marketing, strategy, legal and, of course, technology. These were addressed by real people in real departments who sat down together and worked together. They worked to shape the Open Platform to make sure that it is absolutely right thing to do, from so many points of view.

Matt McAlister brought all the strands together, and must be credited for nurturing and co-ordinating such a major move. But there’s also huge credit to be given to so many individuals within so many Guardian departments who made sure it was a unified, coherent and successful whole. This is a major corporate achievement, but in so many ways a corporation is just another name for individuals working together.

An ABC of R2: Z is for zones

…which are a top navigation level of However the navigational structure was designed to be quite fluid, and the concept of a zone is really more of an internal reference point than a phrase that’s intended to be used by our site’s users.

The ebb and flow of the online media marketplace meant that while we were working on R2 Times Online in the UK launched a redesign of their own. At the time I was interested to read an interview with their Information Architect who said that they had constructed a three-level navigation hierarchy. For we had considered and rejected implementing such an explicit scheme for three primary reasons.

First we recognise that the breadth of our content wasn’t equal across all subjects — we have many more journalists working on news stories than travel stories, for example, so a second-level of navigation under news might have a couple of dozen subjects, which is impossible to display conveniently on a navbar. Second not all subjects break down neatly into three levels — some topics have more depth and complexity than others. Third, Guardian values mean that certain subjects need to be given more prominence than a formal encyclopaedic breakdown allows — for example, in any formal scheme environmental issues would be part of the news or the science section, but the reality is that our planet’s environmental state is far too important to relegate to anything but the highest level.

So R2 was designed to allow fluidity and flexibility in our navigation, and we created words to describe its different aspects. A zone is one of the top-level categories that appear along the top navigation bar: News, Sport, Comment, Culture, etc. At the bottom level of categorisation are very specific subjects such as interest rates, war crimes, Kate Moss and thousands of others. If you’re reading about one of these subjects it doesn’t matter how many levels deep you might be as long as there’s some clear signposting to find your way to your next point of interest. The navigation should give you some kind of orientation, and importantly act as a guide to jumping off to similar subjects or to begin a new journey elsewhere entirely. But exposing any internal classification system is less important. R2 has given us a system that separates the visual navigation from the way content is organised internally.

An ABC of R2: Y is for YAGNI

…which stands for “you ain’t gonna need it” and is an important principle of Agile development, with strong benefits for the business.

The basis for YAGNI stems from a failure common in many software development projects: that when a developer creates a component of a system they tend to give it more flexibility than is immediately necessary, so that it can be reused in more contexts and provide more value. But the result in practice is almost always over-engineering: the component costs more to develop, the suspected additional use rarely materialises, when it does materialise it has requirements that don’t quite match the actual implementation, and the component as a whole is more brittle, less comprehensible to successive developers and more difficult to maintain.

By contrast YAGNI says “you ain’t gonna need it”: develop the absolute minimum to get it working for the immediate problem, and extend it later only if another specific needs arises. This works in Agile development because there is a wider supporting structure which enables changes to be made with minimal cost.

An excellent example of YAGNI in action was our implementation of content workflow. There are lots of things you could do with workflow. You can lock content to ensure that only one person can edit it at a time; you can branch content so that different versions can be worked on simultaneously; one of our senior stakeholders was keen on allowing numbered versions which she had seen elsewhere, allowing individual changes to be tracked and traced, and even allowing someone to switch back to a previous numbered version — but she did say that the feature was almost never used.

What we did in practice was a minimal workflow with minimal cost: we built in the ability to have content as draft and live, and very little more. That was sufficient for the staff who we knew would be using it. It seems like we could have taken the opportunity to build more, in case more people with more needs would use it. But you know what they say — you ain’t gonna need it. In the months following that very early stage in the project the company changed a lot, and among other things this included much more integrated working between editorial teams who were previously designated “web” or “print”… these days that distinction has diminished. The technological impact is that many editorial staff are using an integrated front-end, sitting outside the web system, and that’s where the workflow is handled now.

The industry-wide drive for integrated publishing changed our working environment; if we’d have developed elaborate workflow in the web system it would have been wasted effort, and a large wasted cost for the company.