An ABC of R2: O is for opportunity

…which is a word that we came to understand only slowly, particularly as a counterpart to the word “challenge”.

As we worked we inevitably came across problems; Nigel, our indefatigable programme manager, would insist on calling them “challenges”, and casting possible actions as “opportunities”, to the point that it became a running joke. But problems — sorry, challenges — are chances to raise your game, and opportunities are chances to resolve two issues with one action.

One very early challenge was to deliver our video platform without disrupting the R2 project. Delivering video was itself a major project, requiring CMS integration and embedded advertising. Our opportunity was to do that and at the same time prove that our “business as usual” team (which ran alongside the R2 team and tended to deal with small, one-off tasks) could produce work at least as complex and high-profile as anything the R2 team could.

Our video technology has been a great success. The team produced something which enshrined good principles of web publishing, and integrated perfectly with the content management system (allowing keywording, search findability, etc) built from the R2 project. Taking the opportunity to prove the capability of the business as usual team provided everyone — both inside and outside the team — with even more confidence in what we could do.

An ABC of R2: N is for News section

…which was one of the two highest priority launches of project. Yet it happened around 12 months after we planned it, and between the planning and the launch we also launched the guardian.co.uk home page, video integration, and sections for Media, Technology, Business, Science, Society, Money and Environment. If it was so important, why did we take that seemingly roundabout route?

Actually, it wasn’t that indirect. In January and February 2007 planned all the work that was to follow the launch of the Travel section, which had gone out in November 2006. From our senior stakeholders we sought the business priorities, and there were two major milestones: changing the home page of guardian.co.uk would send the clearest public signal of intent (even though it was only one page), and launching our news content in the new design would demonstrate the depth, extent and utility of the transformation. So those were our two major targets.

But there was another major requirement running through all our launches, and that is that they should be sufficiently comprehensive and largely complete at the moment of launch — there shouldn’t be any obviously missing features or tools. Since the news agenda is both urgent and highly volatile the news desk needed a comprehensive set of tools with strong integration to be able to deal with the daily demand. That included polls, improved galleries, an audio player with better podcast integration, a wider range of layout templates, many more navigational components, more streamlined tools, cartoon pages, very flexible keyword management, and much more.

It was, in total, around a year’s worth of work, so getting there directly would have meant no major launches — no tangible benefits — for a year, and that just wasn’t on. Agile development is about providing value early. To deal with this problem we exploited that fact that many sections, such as Science and Media, would benefit greatly from early launches and wouldn’t need such a comprehesive featureset as News from the word go. And developing those other sections would help build up towards News. We then calculated that if we wanted to get to the News launch with a detour through those other launches then it was a difference of only 6%.

Clearly there were big wins all round. Many sections were launched early so those desks got the benefits early; the commercial team was able to make use of the flexible advertising early; the technology going to the news desk was tested and refined well ahead of launch; and the risks normally inherent in a big bang launch were drastically reduced.

Speaking personally, I found the launch of the News section surprisingly muted. Lots of the public excitement and discussion around the reworking of our site had occurred when we launched the front page many months before, and continued in varying forms with the release of each subsequent section. By the time we got to the News section it was much less startling to those outside looking in. But in terms of internal change so much traffic goes through the News section, and it involves so many people working with it each day, it was very significant. Frankly, big launches which happen with so little fuss is something I’m very happy to live with.

An ABC of R2: M is for milestones

…which are important even on an Agile project.

Many people who read just a little about Agile development think there are no fixed commitments. It’s true that there is constant reprioritisation of work, but that generally operates at the task level, and there is still a need to set goals and stick to them. After all, how else can you give the people who sign the cheques reassurance that you’ll deliver what they want?

So when we planned R2 we also set milestones marking when we would make the major launches of each section. Each milestone was stated very simply at the high level: launch the homepage, launch the Environment section, etc. These were the targets we put before our senior stakeholders, and they were what we were accountable for.

At a lower level we planned for each launch to include specific features: the Culture launch would include a special component to list the critics; the Sports launch would include a component showing the latest betting odds from Blue Square; and so on. It was these lower level details which were always up for reprioritisation. As long as we could deliver sufficient features to allow the launch of each section we could fairly be said to be hitting that milestone and satisfying our senior stakeholders.

An ABC of R2: L is for legacy systems

…which needed to be removed if we were to be productive. Just adding shiny new things would only add to our workload if we didn’t also get rid of the old ones.

In fact the most significant legacy systems needed to be removed (or at least isolated) before we could even start R2. These included internal dependencies on an unsupported browser, a cumbersome (and imminently unsupported) application server, and a particularly insidious database table.

Eliminating these was not a glamorous job by external standards, but it was a necessary one. And significantly it allowed us to prove and improve our Agile skills. Each project was more complicated than the one before. By the time we got to the last of these we believed we really could undertake the largest software project since the website began.

An ABC of R2: K is for keyword component

…which was the first visible feature we released as part of the R2 project. That was way back in May 2006, and it appeared on articles in the Travel section.

The keyword component was simply a box listing keywords associated with the article, but to get even to that modest point was a long journey. Here are some of the things we needed to build to get there.

  • A continuous integration pipeline, which automatically took the software at any time from source code to deployable bundle via automated testing and reporting;
  • A tool to create and manage keywords — some keywords would need to take the user to specific landing pages, others to a page of search results;
  • A tool to add keywords to articles — the tool running in our new system, the article editor running in our old system, and the two integrating seamlessly;
  • A manageable abstration layer at the web server level that allowed pages from the legacy content management system to include components from the new R2 system.

Naturally this component didn’t change the world, but it did provide validation for our entire development pipeline. And the component survives today, in a more sophisticated form, on all our content, where you can use it to navigate to related subjects.

An ABC of R2: J is for JFDI

…which stands for “just do it”, and was the unofficial name of one of the development teams which sat alongside the R2 teams.

One key principle we had from the start of the project was that other development work couldn’t stop for the sake of the site rebuild. There might be less of it, but it shouldn’t dwindle to zero. And while R2 was a major long term undertaking, the rest of the work that came up inevitably had a very different shape. Consequently we had different kinds of teams.

The JFDI team handled very short turnaround work. Mostly this consisted of bugfixes, but it also included minor enhancements. It worked in a traditional Agile manner, but due to the size of the individual tasks work was reprioritised every day rather than every fortnight.

Working on the JFDI team suits some people better than others. On the one hand it’s difficult to get your teeth stuck into anything because it doesn’t last very long (or at least it shouldn’t); on the other hand you get a sense of completion every day. A lot of the time people don’t relish cycling into that team, but once they’re in they find they learn a huge amount about how the software they’ve written actually gets used. I’ve written more on this subject previously.

Overall the JFDI team has been very successful, dealing with a large and constantly-shifting workload, but also demonstrating daily progress to our internal users. Since R2 finished we’ve kept the team running in the same mode, and it continues to bring immediate benefit to people inside and outside the development teams.

An ABC of R2: I is for iterations

…which lasted two weeks and culminated in a full new release of our software. I’ve written elsewhere about what happened in one week of a particular iteration in June 2008.

However, our R2 iterations didn’t just involve implementing software. At the same time each team was also working with a business analyst and end-users to plan and clarify the work for the next iteration, and sometimes getting the last iteration’s work through its systems tests prior to release. As one of the team leads said to me, “we work in three timezones”.

At the end of an iteration the release went out. But our releases didn’t always reveal much that was different — either because we were waiting to reveal it as part of a forthcoming launch, or because it was part of our internally-facing systems. That can be an odd feeling for the development teams. In previous lives I’ve worked on projects where the launch has the whole team working right up to the last moment and so also signals the time when you can collapse with exhaustion. But our interleaving of “timezones”, and releases well ahead of launches, changed that. The launch of, say, the Sports section was a big moment for so many in the company, but most of the people on the development team had finished the work some time before and were spending more time thinking about the launch that followed.

An ABC of R2: H is for home page

…which was launched in May 2007 and incorporated a huge amount of flexibility to tell the day’s news in different ways.

There are two major aspects to the home page’s flexibility. The first, and most obviously, is a variety of templates. In our previous system the home page had almost no flexibility at all, which was a consequence of not separating the content from the presentation — the home page was effectively a small program and changing the layout meant changing the code. That’s not something you can do in the middle of busy news day. As part of the R2 project we created a variety of templates which could be switched in largely at will. I say “largely” at will, because switching layout also means the various areas of the page are spaced differently, some growing and others shrinking, so you need to make sure they all have the right amount of content in them.

The second kind of flexibility is in what you can do on the page. Any one template has internal logic which changes the layout subtly according to where a production staffer marks a break or places an image or video.

We launched our new home page early on 10 May 2007. Around midday Tony Blair announced he was stepping down as prime minister. The event gave the news desk the chance to make use of many alternative templates in various configurations. In the tech team we would check back at the home page throughout the day to see yet another template was on show for the first time. By the end of the day our new software had been given more of a workout than we could have guessed, but it served us all admirably.

An ABC of R2: G is for Guardian America

…which was not part of the project scope when we started R2. It’s fair to say that when we began implementing in February 2006, the idea of a Guardian America launch was not on the radar. Yet by the middle of 2007 it was being talked about very seriously, and increasingly so. How did we fit in an additional sub-project?

As much as technologists might sometimes think they hold the key to success, when it comes to media it’s still true that content is king. Guardian America’s success is driven from its editor, Michael Tomasky, and his team. But technology did provide the vehicle for that.

The most valuable thing we technologists did for Guardian America was to not reinvent the wheel. We recognised that the core elements had already been built — most notably a front page designed to showcase a variety of content was already in use as the guardian.co.uk front. Making use of that also ensured design consistency. But that’s not to say there was no work to do. The content management system was not originally designed to support two major fronts, particularly with a variation in branding. The core work, then, was to extend something we had already delivered and make it more useful.

Using this approach everyone won. The Guardian America team got all the functionality and flexibility that we had delivered for the team running the original site front, and they got it relatively quickly. From an operational point of view, by managing Guardian America in the same way the guardian.co.uk front was managed, the GA team was able to share skills, people, advice and creative ideas. The tech team got to show they could deliver technology for a high-profile project on very short timescales. The business as a whole won because the overall R2 budget remained constant. We managed that by deprioritising a small amount of less important work in the usual Agile manner. I don’t think there were any arguments about what, exactly, was deprioritised, since Guardian America was so clearly more significant than many other features on our list.

An ABC of R2: F is for flexible advertising

…which was one of the key goals of the project. This exposes one of the significant aspects of R2: it was neither editorially driven, nor technically driven, nor commercially driven. It was driven by a unity of needs right across the company, and it needed to be successful in all these areas.

There are a few ways in which we’ve added flexibility to our advertising system, but I’ll mention just a couple here.

One form of flexibility is in the use of our ad slots. Our pages have been designed to display ads of various sizes, and reshape themselves accordingly. This is most obvious on some of the right hand ad slots — sometimes you’ll find them displaying an ad that’s a squareish rectangle, sometimes you’ll return to the same page and find another that’s markedly taller. In theory that’s a trivial piece of work, and HTML and CSS can handle that with ease. But in practice, of course, it’s more challenging. One of those challenges is trying avoid using an iframe container which ensures ads don’t unduly interfere with the rest of the page, but which force you define their size before the ad server has had a chance to tell what size the ad needs to be.

Another, related, form of flexibility is in our non-use of ad slots. You’ll be able to see this on the guardian.co.uk home page which sometimes has ads on it, and sometimes doesn’t. In the early stages of R2 the ad sales team made a decision I’ve always admired: not to fill certain ad slots just because they could, and ensure there were periods when the guardian.co.uk home page was ad-free. Apart from any other advantages this means that when ads do appear they have more impact, and of course it increases the value of advertising on the home page.

I like the work we’ve done on advertising less because of the technical achievements, but much more because it shows we’ve used R2 as a chance to rethink how we approach our business and break away from some conventional thinking.