Category: Guardian.co.uk

An ABC of R2: U is for URLs

·

…whose structure we worked hard to get right as part of the project. This was an important part of weaving ourselves into the fabric of the web: to ensure our referencing system had a useful structure to those outside our organisation.

Previously our URLs looked like this:

http://politics.guardian.co.uk/tonyblair/story/0,,2075005,00.html

today the same piece of content is referenced like this:

http://www.guardian.co.uk/politics/2007/may/10/tonyblair.labour3

There are two obvious changes here. First, the obscure numbers have gone — a legacy of our old Vignette StoryServer software, and utterly useless to anyone who happens to be a human being. Second, we’ve included the date in a friendly format, to provide some sense of orientation, and particularly useful if you happen to be looking at several URLs at a time.

There are other changes, too. The “politics” reference has come out of the domain name and into the path, to reflect the fact that all our content is part of the guardian.co.uk site. A couple of keywords also appear in the URL, partly to convey what the content is about, and partly to distinguish it from any other content that appeared in the same section on the same day. As it happens, quite a lot was written about Tony Blair and Labour on that day, so the “3” adds a further unique identifier.

In some ways URL structure is an obscure topic, but getting it right opens up access to more of our content in more ways to more people. Further examples have been given by Matt McAlister in his discussion of the RSS feeds.

An ABC of R2: T is for timeline

·

…which looked like this:

  • October 2005: Start detailed planning of Travel section. Involved about 12 people from Guardian and ThoughtWorks. Much preparation needs to take place before work can begin.
  • February 2006: Start building the Travel section, initially focusing on the servers, development environment and build pipeline.
  • May 2006: Launch first visible item on Travel section, a keyword component.
  • November 2006: Launch entire Travel section. From a technical standpoint it was then the biggest single launch in the website’s history, and one of the smoothest then known.
  • January 2007: Start planning phase two. Involved around 20 people fulltime for six weeks, but sufficient clarity was achieved after three weeks to start one stream of work then.
  • May 2007: Launch new guardian.co.uk home page. Within hours of the launch, Tony Blair obligingly announces his resignation as prime minister, allowing us to make full use of our flexible templates as the story develops throughout the day.
  • July 2007: Launch Science, Technology and Environment sections. Included a new feature: series. This allows special navigation back and forth through items in a series such as Bad Science or Nature Notes.
  • August 2007: Launch integrated video. Includes reskinned Brightcove player with dynamically-inserted pre-roll advertising.
  • October 2007: Launch sections for Business, Society, Media, Money and Guardian America.
  • February 2008: Launch Observer, News, and Politics sections, plus integrated audio.
  • April 2008: Launch Sport and Football sections. Includes live scores and fixtures, and match facts among other features.
  • June 2008: Launch Comment is Free. Includes community features from Pluck.
  • August 2008: Launch of Culture, Education and Life & Style sections, plus 14 blogs moving over from the old Movable Type system.
  • September 2008. Launch the remaining 21 blogs onto the new platform. Some of these blogs didn’t go in August due to the key part they played in August news events, and some because they were awaiting additional features such as remote working tools.

An ABC of R2: S is for sitebuilding

·

…which was the penultimate step before a launch, after the software had been built and released, and before the technical work to finally lift the curtain.

One of the big changes that was part of R2 was how we structured our content — our information architecture. Previously each piece of content lived in a section, up to two levels deep, and a lot of content was duplicated so that it could appear in more than one section at a time. An extreme example we often used was the affair around David Kelly and the consequential Hutton inquiry. Almost every story there crossed the boundaries of politics, media and daily news.

With R2 we were introducing much more nuanced keywording and more options around navigation. So the content in the old system didn’t map directly into the new system: it all had to be examined and reclassified by hand. Additionally, production staff needed to build subject pages in ways they couldn’t before — for example, the pages on Afghanistan, the British monarchy and the BAE corruption investigations. All this was called sitebuilding.

Of course, the tech team built tools and wrote scripts to make the production staff’s job easier, but some things just need human expertise and take a very long time. Typically we allowed six weeks between the time the software was released and the relevant site was launched and that was the period in which sitebuilding took place.

An ABC of R2: R is for R2

·

…which people often think stood for the fact that we were building “revision 2” or “release 2” of guardian.co.uk. It didn’t stand for that, not least because this is actually the third or fourth such revision since we launched in 1999. In fact, it stood for “rebuild and redesign”.

However, while plans for the project were still being hatched, and for just a very short time, it had another name. We were just finishing a day planning the project’s priorities and were aware that when The Guardian newspaper changed to its Berliner format the transformation was called internally Project Kennedy. With slightly too much levity at the end of a long day in single room we chose another 60s icon to provide the name for this project. At the time it seemed like a jolly good idea.

Next morning the cold, fluorescent office light of reality hit: no-one wanted to go in front of the board and ask for a large amount of money for something called that. After a flurry of e-mail Mina came up with R2, and everyone liked that — it was neutral and serious.

So R2 it was, and remained.

An ABC of R2: Q is for quality assurance

·

…which is a much misunderstood subject.

The R2 team consisted of a number of QAs, and the most obvious artifact that the QAs produced and worked with was the test script: a series of detailed instruction that explained what to test and how to test it. For this reason it’s too easy to dismiss QAs as testers, and that would be a mistake.

Our QA Manager, Amy, says “I don’t want us to be finding problems, I want there to be no problems to find”. That hints at how QAs should be used: they should be involved from the very start of a piece of work to identify an appropriate structure and approach that ensures greater reliability and more opportunities for testing. In the best cases a QA has guided a task in a direction that BAs, developers and architects might not have previously considered, so avoiding problems they hadn’t thought of. In the worst cases a QA has been omitted from planning a piece of work and something considered straightforward by the QA-less group has turned out expensive to test and a constant source of problems.

Testing comes at the end of the development process, and considering a QA as a tester therefore allows one to fall into the trap of including them only at the end. The quality assurance process should add value, and that can only happen if the QAs are involved from the start.

An ABC of R2: P is for pair programming

·

…which was, and is, a hugely important part of our software development, and something that took a long time to learn to do well.

Pair programming is when two developers sit at one machine and one keyboard to write the software. It’s very difficult to do: the driver has the pressure of someone watching their every move, and the navigator has to be aware of what’s going on because they’ll be asked to take over at any moment and they have a responsibility to keep an eye on the bigger picture. It also makes it a very collaborative process — the pair need to work out together exactly how they’re going to tackle every problem. Mat, leading our architecture team, calls this “keeping each other honest”.

Pairing looks expensive — two people apparently doing something that one could do — but that makes the mistake of thinking that all software is the same and all developers are interchangeable. Here are some of the benefits we’ve found:

  • Developers are more productive. With someone sitting beside you you can’t afford to cruise. You’ve got to be demonstrably on the ball all the time.
  • The quality of the software is much higher. I’ve listened to developers discussing how they should write a particular piece of code, suggesting alternatives and weighing up pros and cons that one individual would never have come up with by themselves.
  • Developers become much more skilled much more quickly. Everyone learns off everyone else.
  • The company’s technology investment is de-risked. Highly specialised knowledge is shared among many people, and doesn’t live just in the head of one person. This also means…
  • …Resourcing projects is easier. Because more people are able to move onto other teams more often, since (a) they are more likely to have the knowledge needed for the new team, and (b) they are less tied to their existing team since they will have shared that knowledge. This last point also means…
  • …Developers have more opportunities to learn new things. They can move onto other teams and new projects, safe that they won’t be constantly called back to their previous team, because they’ll have shared their skills and knowledge.

When a software project is complete the software itself is only just beginning its life, in operation day after day — and in the case of our software, by hundreds of people for many years to come. So that development investment has to ensure the product is of very high quality, and pair programming is part of how we ensure that.

An ABC of R2: O is for opportunity

·

…which is a word that we came to understand only slowly, particularly as a counterpart to the word “challenge”.

As we worked we inevitably came across problems; Nigel, our indefatigable programme manager, would insist on calling them “challenges”, and casting possible actions as “opportunities”, to the point that it became a running joke. But problems — sorry, challenges — are chances to raise your game, and opportunities are chances to resolve two issues with one action.

One very early challenge was to deliver our video platform without disrupting the R2 project. Delivering video was itself a major project, requiring CMS integration and embedded advertising. Our opportunity was to do that and at the same time prove that our “business as usual” team (which ran alongside the R2 team and tended to deal with small, one-off tasks) could produce work at least as complex and high-profile as anything the R2 team could.

Our video technology has been a great success. The team produced something which enshrined good principles of web publishing, and integrated perfectly with the content management system (allowing keywording, search findability, etc) built from the R2 project. Taking the opportunity to prove the capability of the business as usual team provided everyone — both inside and outside the team — with even more confidence in what we could do.

An ABC of R2: N is for News section

·

…which was one of the two highest priority launches of project. Yet it happened around 12 months after we planned it, and between the planning and the launch we also launched the guardian.co.uk home page, video integration, and sections for Media, Technology, Business, Science, Society, Money and Environment. If it was so important, why did we take that seemingly roundabout route?

Actually, it wasn’t that indirect. In January and February 2007 planned all the work that was to follow the launch of the Travel section, which had gone out in November 2006. From our senior stakeholders we sought the business priorities, and there were two major milestones: changing the home page of guardian.co.uk would send the clearest public signal of intent (even though it was only one page), and launching our news content in the new design would demonstrate the depth, extent and utility of the transformation. So those were our two major targets.

But there was another major requirement running through all our launches, and that is that they should be sufficiently comprehensive and largely complete at the moment of launch — there shouldn’t be any obviously missing features or tools. Since the news agenda is both urgent and highly volatile the news desk needed a comprehensive set of tools with strong integration to be able to deal with the daily demand. That included polls, improved galleries, an audio player with better podcast integration, a wider range of layout templates, many more navigational components, more streamlined tools, cartoon pages, very flexible keyword management, and much more.

It was, in total, around a year’s worth of work, so getting there directly would have meant no major launches — no tangible benefits — for a year, and that just wasn’t on. Agile development is about providing value early. To deal with this problem we exploited that fact that many sections, such as Science and Media, would benefit greatly from early launches and wouldn’t need such a comprehesive featureset as News from the word go. And developing those other sections would help build up towards News. We then calculated that if we wanted to get to the News launch with a detour through those other launches then it was a difference of only 6%.

Clearly there were big wins all round. Many sections were launched early so those desks got the benefits early; the commercial team was able to make use of the flexible advertising early; the technology going to the news desk was tested and refined well ahead of launch; and the risks normally inherent in a big bang launch were drastically reduced.

Speaking personally, I found the launch of the News section surprisingly muted. Lots of the public excitement and discussion around the reworking of our site had occurred when we launched the front page many months before, and continued in varying forms with the release of each subsequent section. By the time we got to the News section it was much less startling to those outside looking in. But in terms of internal change so much traffic goes through the News section, and it involves so many people working with it each day, it was very significant. Frankly, big launches which happen with so little fuss is something I’m very happy to live with.

An ABC of R2: M is for milestones

·

…which are important even on an Agile project.

Many people who read just a little about Agile development think there are no fixed commitments. It’s true that there is constant reprioritisation of work, but that generally operates at the task level, and there is still a need to set goals and stick to them. After all, how else can you give the people who sign the cheques reassurance that you’ll deliver what they want?

So when we planned R2 we also set milestones marking when we would make the major launches of each section. Each milestone was stated very simply at the high level: launch the homepage, launch the Environment section, etc. These were the targets we put before our senior stakeholders, and they were what we were accountable for.

At a lower level we planned for each launch to include specific features: the Culture launch would include a special component to list the critics; the Sports launch would include a component showing the latest betting odds from Blue Square; and so on. It was these lower level details which were always up for reprioritisation. As long as we could deliver sufficient features to allow the launch of each section we could fairly be said to be hitting that milestone and satisfying our senior stakeholders.

An ABC of R2: L is for legacy systems

· 3 Comments ·

…which needed to be removed if we were to be productive. Just adding shiny new things would only add to our workload if we didn’t also get rid of the old ones.

In fact the most significant legacy systems needed to be removed (or at least isolated) before we could even start R2. These included internal dependencies on an unsupported browser, a cumbersome (and imminently unsupported) application server, and a particularly insidious database table.

Eliminating these was not a glamorous job by external standards, but it was a necessary one. And significantly it allowed us to prove and improve our Agile skills. Each project was more complicated than the one before. By the time we got to the last of these we believed we really could undertake the largest software project since the website began.