Sometimes a piece of work is so big that it’s quite reasonably called not a “project” but a “programme”. And at the top level of the programme is a group … Continue reading Healthy programme-level conversations
Category: Guardian.co.uk
I’m leaving the Guardian
I’m leaving the Guardian. It’s a sad time for me, but also one of great pride because I’ve got a chance to look back on eight years of digital growth and recall some of the wonderful things we’ve done…
Early days
I introduced agile development to the Guardian team in 2003. I was not particularly an advocate of agile before then, but I saw it addressed many of our needs — the need to continually evolve our technology, to spread knowledge within the team, to deliver more predictably, and more.
But I couldn’t have done that without the trust and support of Liz Sarginson and Stephen Dunn, who were the ones who brought me into the organisation. I pitched it to them, and they could have said no. Or they could have said “Yes, but”. But they didn’t — they gave me their trust and support, and we made it work, although it was difficult — particularly for the developers, who need to deal with then-challenging ideas such as test-driven development with legacy code, and pair programming.
The trust they extended to me then has stayed with me since, and it’s something I try to remember when people come to me with their ideas now.
Slaying dragons…
It was the development team that identified our greatest challenges and convinced me that we — or, rather, they — could take them on. Those challenges were eliminating our most resource-draining legacy systems.
The first was updating our CMS tools to work in a more modern browser. At the time the tools were tied to Netscape 4, which came out in 1997. The project turned out to be much bigger than we’d guessed, because our own tools weren’t documented and we kept finding obscure features which seemed essential for someone, somewhere. But we managed it without compromising our ability to deliver shorter-term, more immediate things along the way.
The second was eliminating a database table. The table itself was originally introduced as a simple performance optimisation, but it had grown unexpectedly over the years, and it fired so many triggers that it was now making some of our tools so slow as to render them useless. But references to the table were scattered all over the vast codebase. It was the work of months to eliminate all references to it and finally to drop it. But we did it, without impacting any other work, and we popped a champagne bottle open when our DBA successfully typed DROP TABLE… and the prompt returned without complaint.
The third was the elimination of our legacy CMS engine, which ran our site’s thousands of templates and scripts. Over the years it had been configured and customised far beyond the vendor’s expectations and no longer fitted their roadmap. Also, it denied us reliable release mechanisms and modern version control for the code we put in it. We replaced it with an open source alternative which still ran the same templates and scripts. The migration was gradual, steady, and every step had a rollback plan.
…and building castles
After slaying those dragons of the past we were able to build for the future. We took on our biggest ever project, dubbed R2, to rebuild and redesign guardian.co.uk. It introduced much more flexible templates and content management for the editorial teams, more flexible advertising options for our commercial teams, and more modern technologies and processes for the technical team. It lasted two and a half years, delivered new code into production reliably every fortnight, peaked at over 100 people, and came in on time and under budget.
It was also designed and built with technical integrity: our mantra was to be “of the web, not just on the web”. In the many months that followed we released a full-content API, a plugin framework, a new commenting system, and innumerable other products and features, large and small.
It’s the people, stupid
But beyond all the things we did, it’s the people I will remember with greatest pride. It was the team that stepped up to the mark and slayed the legacy dragons; it was people across the organistion who made our major rebuild and redesign a thing for the future; and it is trust and strong relationships within and without the technology team that allow ideas to continually flourish.
All long-standing media organisations face an uncertain future. If any of those can see it through then it’s the Guardian, primarily because of its incredibly talented people.
Agile Awards 2011 shortlist
I returned from last week’s holiday to an e-mail telling me that the Guardian has been shortlisted in the 2011 Agile Awards. In not one, but two categories: Best Agile Team, and Best Use of Agile in the Private Sector. I’m thrilled, because it acknowledges that success really is a team effort, and while you’re in the middle of the maelstrom it’s difficult to see if you’re making any kind of difference. Clearly the Guardian team has been recognised.
I should also blushingly acknowledge that I’ve been shortlisted for Agile Project or Programme Manager of the Year. “Blushingly” because the crowd I’m among is great indeed. Congratulations to all those people, organisations, and teams.
The awards dinner is on Wednesday 5 October, in London, and if it’s anything like last year’s event it will be a terrific night. And if that wasn’t enough, it also benefits two charities.
Guardian Hack Day 2
There isn’t too much I can add to Kevin Anderson‘s extensive coverage of Guardian Hack Day 2 [1, 2, 3, 4, 5, 6]. But here are a couple of things…
First, huge congratulations to Matt McAlister and the whole organising team. Hack Day 1 set the bar pretty high, but the team managed to meet that, if not surpass it.
Second, a terrific change from the last one was that so many people seemed to be sharing skills and ideas across organisations. People from Tinker.it, the BBC, Inkling, Google, Talis, Omniture, Rabbit, Guardian and elsewhere all mixed it up to produce some really innovative stuff. Added to this, there was a healthy number of people who don’t claim to be programmers creating some very interesting hacks, thanks in part to Simon Willison’s list of Hack Day tools for non-developers.
So among other things we got a tracker of the most controversial articles, a page of search terms that people are using to find guardian.co.uk — live — a social network for dead people, and much, much more.
The photo here shows Matt under the gaze of the Guardian’s inspirational editor CP Scott — or rather a bust of him. I’ve put this photo and others in a Flickr photo pool. If you were there, please feel free to contribute.
The Open Platform: It’s thanks to individuals
Yesterday we launched the Guardian Open Platform, incorporating the Content API and the Data Store. A lot’s been said already about this — and more will be said — so I will add only a few words of acknowledgement. Many have pointed out (Tom Watson, Jeff Jarvis, ReadWriteWeb, Fast Company and others) that it’s a serious move by the company itself. So I really want to spend a moment to reflect on what that really means.
It can really be best captured by the questions asked by the crowd at the launch event: What if other news organisations want to reuse your journalism? Will the ad network really work? Do the numbers add up? These and other questions were asked at the time and were asked inside the organisation in the months leading up to the launch. The questions span journalism, advertising, finance, technology, marketing, strategy, legal and, of course, technology. These were addressed by real people in real departments who sat down together and worked together. They worked to shape the Open Platform to make sure that it is absolutely right thing to do, from so many points of view.
Matt McAlister brought all the strands together, and must be credited for nurturing and co-ordinating such a major move. But there’s also huge credit to be given to so many individuals within so many Guardian departments who made sure it was a unified, coherent and successful whole. This is a major corporate achievement, but in so many ways a corporation is just another name for individuals working together.
An ABC of R2: Z is for zones
…which are a top navigation level of guardian.co.uk. However the navigational structure was designed to be quite fluid, and the concept of a zone is really more of an internal reference point than a phrase that’s intended to be used by our site’s users.
The ebb and flow of the online media marketplace meant that while we were working on R2 Times Online in the UK launched a redesign of their own. At the time I was interested to read an interview with their Information Architect who said that they had constructed a three-level navigation hierarchy. For guardian.co.uk we had considered and rejected implementing such an explicit scheme for three primary reasons.
First we recognise that the breadth of our content wasn’t equal across all subjects — we have many more journalists working on news stories than travel stories, for example, so a second-level of navigation under news might have a couple of dozen subjects, which is impossible to display conveniently on a navbar. Second not all subjects break down neatly into three levels — some topics have more depth and complexity than others. Third, Guardian values mean that certain subjects need to be given more prominence than a formal encyclopaedic breakdown allows — for example, in any formal scheme environmental issues would be part of the news or the science section, but the reality is that our planet’s environmental state is far too important to relegate to anything but the highest level.
So R2 was designed to allow fluidity and flexibility in our navigation, and we created words to describe its different aspects. A zone is one of the top-level categories that appear along the top navigation bar: News, Sport, Comment, Culture, etc. At the bottom level of categorisation are very specific subjects such as interest rates, war crimes, Kate Moss and thousands of others. If you’re reading about one of these subjects it doesn’t matter how many levels deep you might be as long as there’s some clear signposting to find your way to your next point of interest. The navigation should give you some kind of orientation, and importantly act as a guide to jumping off to similar subjects or to begin a new journey elsewhere entirely. But exposing any internal classification system is less important. R2 has given us a system that separates the visual navigation from the way content is organised internally.
An ABC of R2: Y is for YAGNI
…which stands for “you ain’t gonna need it” and is an important principle of Agile development, with strong benefits for the business.
The basis for YAGNI stems from a failure common in many software development projects: that when a developer creates a component of a system they tend to give it more flexibility than is immediately necessary, so that it can be reused in more contexts and provide more value. But the result in practice is almost always over-engineering: the component costs more to develop, the suspected additional use rarely materialises, when it does materialise it has requirements that don’t quite match the actual implementation, and the component as a whole is more brittle, less comprehensible to successive developers and more difficult to maintain.
By contrast YAGNI says “you ain’t gonna need it”: develop the absolute minimum to get it working for the immediate problem, and extend it later only if another specific needs arises. This works in Agile development because there is a wider supporting structure which enables changes to be made with minimal cost.
An excellent example of YAGNI in action was our implementation of content workflow. There are lots of things you could do with workflow. You can lock content to ensure that only one person can edit it at a time; you can branch content so that different versions can be worked on simultaneously; one of our senior stakeholders was keen on allowing numbered versions which she had seen elsewhere, allowing individual changes to be tracked and traced, and even allowing someone to switch back to a previous numbered version — but she did say that the feature was almost never used.
What we did in practice was a minimal workflow with minimal cost: we built in the ability to have content as draft and live, and very little more. That was sufficient for the staff who we knew would be using it. It seems like we could have taken the opportunity to build more, in case more people with more needs would use it. But you know what they say — you ain’t gonna need it. In the months following that very early stage in the project the company changed a lot, and among other things this included much more integrated working between editorial teams who were previously designated “web” or “print”… these days that distinction has diminished. The technological impact is that many editorial staff are using an integrated front-end, sitting outside the web system, and that’s where the workflow is handled now.
The industry-wide drive for integrated publishing changed our working environment; if we’d have developed elaborate workflow in the web system it would have been wasted effort, and a large wasted cost for the company.
An ABC of R2: X is for XL
…which was a size of problem we noted but wouldn’t tackle.
When we estimated the work for R2 up front we used t-shirt sizes for each feature: S, M, L and XL. The largest single task the team would tackle was an L, which was the equivalent of five days’ work. We felt this was a good maximum for two reasons: first, it delivered something of value within a reasonable space of time and what should have been one iteration; second, if we could imagine developing it within five days then it was probably sufficiently understood, whereas anything bigger risked being too complex to estimate without a lot more thought.
However, during the time-limited initial planning period we were required to estimate things that needed more analysis than we had time for. These features we labelled XL and gave a nominal equivalent size to: 10 days’ work.
However, we would never actually start a task that was labelled XL. We would always break it down before the event into smaller parts. The aim was that although individual estimates might be over or under, on average they should balance out and the total size of the project should remain constant.
Of course, for XL tasks it was quite possible that a more thoughtful estimate would produce a total that was far too big to be able to be balanced out by other tasks. In these cases we would have to reconsider our options and think about dealing with the problem in a different way — most likely by having a less comprehensive solution. However that didn’t tend to be too much of a problem. In many cases by the time we got to the XL task our stakeholders’ needs and priorities had changed so much that there were a lot of new options and directions that people wanted to explore, and ones that they would not have thought of in those very early planning stages.
This is an example of why planning was important, but very detailed planning would actually have been quite wasteful.
An ABC of R2: W is for Wiimbledon
…which was a semi-regular event of Wii tennis in the office, but a very useful part of our R2 work, too.
Each launch required a small army of technologists to be on-hand: to run the various scripts, check the results, and deal with any problems that might arise. We needed to arrange these teams carefully because launches happened overnight, so we’d need an overnight team and another team in early the next day to pick up any remaining issues.
All this was fine, but after a while someone realised we’d missed a trick. I think he actually wanted to be part of the overnight team (it’s always exciting to see these thngs go live) but wasn’t actually on it. So he arranged with others to bring in some games consoles, and wired one up to a big screen, the others to projectors. It was a big draw, and a great way to have an extra group of techies staying late in case the need arose.
Still, we couldn’t be cavalier with this. For example, we made sure the gamers were located far from the launch team — the launch team had a serious job to do and didn’t need distraction. But that didn’t prevent the benefits: every so often a gamer would slip out of Guitar Hero and wander over to the launch team to check up on progress, offering some advice and support if necessary. One time a critical SQL script was running worryingly slowly and a call came through to see if someone could contribute to the investigation; a couple sat down at a machine a few feet from the Mario Kart players, traced the problem, and suggested a change to the script which was agreed and did the job. That night it made the difference between “go” and “abort”.
If there’s a general lesson to be drawn from this I don’t think it’s to keep a Wii console in the stationery cupboard along with the paperclips and envelopes. But exploiting opportunities that are specific your particular situation is probably a good thing to do, even if they aren’t enshrined in official company policy.
An ABC of R2: V is for video content
…which changed in the way we thought about it (and implemented it) a long time after it was released.
Originally any piece of content was designated as being of exactly one type: it was either an article, a video, a competition, and so on. To take the video content as an example, putting a video onto the site involves opening up a video content editor in the CMS, referencing the appropriate asset in the Brightcove system, and adding some video-specific metadata such as its source and preferred display size. Clearly you aren’t going to confuse that with writing an article, which is why it made sense to say that each piece of content was of exactly one type, no more and no less.
But one day one of the editorial staff made an unusual request: she had an article and wanted us to add a checkbox in the article editor marked “Display on video page”. This puzzled the tech team initially — if it was an article then people wanting videos wouldn’t want it to appear on the video page. Also, we were wary about adding a feature which would complicate the article editor but be used by few people. We quickly found the good reason for her request: she was creating an article with an embedded YouTube video, and the purpose of the content really was to showcase the video. Anyone looking for videos on the site would want to find the content, so it should appear on the video page, but the video should remain outside the Guardian CMS, so the video content editor wasn’t appropriate.
We realised this was a general need: content could fall into several categories at once and editorial staff should be able to flag it as such. The solution, then, was not to add a checkbox marked “Display on video page” because that wouldn’t solve the general problem. Instead we took an existing “labelling” facility present in all the content editors, and we extended it allow content-type labels to be added. Of course, we also needed to ensure the system responded to these new labels, but since it was an extension of an existing feature it was a more natural and reliable solution than we might otherwise have implemented.
The end result therefore didn’t introduce any complexity for end-users — no extra buttons or checkboxes — but it did give them the power to do what they wanted.
This is a good example both of how our understanding of apparently-basic questions (“what is video?”) can change with experience, and of how understanding the source of a requirement can lead to better solutions.
niksilver.com 
