Guardian.co.uk: Steps to a smooth launch

At the weekend the Guardian website went through one of the most significant transformations in its history: we moved our news, politics and Observer content into the new design and new content management system, and we simultaneously launched a lot of new functionality, both internal and external.

There’s an introduction and discussion on the more public-facing aspects of this, kicked off by Emily Bell. For my part, I want to talk briefly about one of the most remarkable behind-the-scenes aspects of it: how we got the weekend launch to go so incredibly smoothly.

The secret is that the weekend’s work was only the final step after a great many before it, all of which were safely out of the way before the weekend…

Guardian.co.uk global navigation bar1. Software release

The actual software was released some weeks ago, in early January. This means that by the time of the launch it had been in use for some time, almost all the lines of code having been executed several hundred (and in some cases thousand) times already, in the production environment.

Even that January release was only an enhancement of previous releases which have been going out fairly quietly over the previous few months. The latest one included new internal tools, and updates to tools, to support some of the new features that are visible today.

Guardian and Observer archive navigation2. Building the pages

Meanwhile editors, subeditors and production staff have had to learn to use those tools. They’ve been using them to migrate a lot of the older content into the new-format pages. You might think that could be done by machine, but that’s not the case. Since we’re also changing our information architecture — adding a lot of semantic structure where previously there was only presentational information — it takes an informed human being to know how to convert an older page into its newer format. A real person is needed to understand what a piece of content is really about, and what it isn’t about but merely mentions in passing. We also need people to look at the new keyword pages (for example, the one on immigration and asylum, or the page on Kosovo), which are populated mostly automatically, and for them to highlight the most important content: the most important content won’t necessarily be the newest.

This work had been going on for many weeks before the weekend launch. The January software release brought in some tools refinements to help them tidy up final loose ends (and no doubt some more tidying will happen over the next couple of weeks).

You can see from this that it’s about much more than mere software. The software enables people to do the editorial work, and the editorial work has been going on for some considerable time. Everything they’ve done has also been tested and previewed, which allows them to see what our external users would see were it to go live. Again, this exercises the software fully, but only within the internal network, before it’s exposed to the outside world.

3. Rehearsals

The work for the weekend launch is mainly running a lot of database scripts to make various new URLs live and decommission old ones. The reason this is such a huge launch is that there’s over ten years’ worth of news content to expose, as well as new functionality and designs.

We couldn’t trust the scripts to work first time (of course), so we spent a lot of time copying the production content into a safe environment, and rehearsing the process there, with real data. We needed to be sure not just it would work, but also how long it would take (considering any hardware differences), and change the scripts and process accordingly.

Guardian.co.uk favicon4. Launch

Finally, after all the rehearsals, the real deal. The work to run the database scripts and raise the curtain on various features ran over Saturday and Sunday, but it was calm enough and organised enough that the team needed to work only slightly extended working days.

So the big launch was a culmination of a huge amount of effort, and the weekend work was after an awful lot of practice. There were a couple of sticky moments, but nothing the team couldn’t recover from in a few minutes. As one of the developers remarked towards the end of Sunday: “The fact that today’s been really tedious is a good thing.”

What we can see now includes

What’s next…

We’ll be ironing out a few things over the next few days, but everything’s gone to plan for now. And then, as Emily says, there’s still sport, arts, life and style, and education to do.

The unstoppable urgency of web development

While I’m usually proud of the work I’m involved with, I’m rarely happy for long. There are always ways to improve, and I’m usually dissatisfied by one unmet ideal or another. Almost since I started in this field I’ve been vexed by how much of web development is “urgent” rather than “important”. This is not merely of theoretical interest; it vexes me because dealing with important issues gets you to something of lasting and strategic value, while dealing with urgent issues gets you to the next day. It’s that unmet ideal of achieving more long-lasting value that causes me to think about this.

I want to give some context around the claim that there’s more urgency in web development, and then I want to offer some ideas about why this might be. If you want the brief version, I think what makes the web generate such urgency is that…

  1. Updates are cheaper;
  2. Problems hurt less;
  3. Competition is more visible;
  4. Boundaries are blurred; and
  5. It’s easier to have more stakeholders.

But first, some context.

Picture will go here when I have timeSome context

Although I do think web development has a surfeit of urgency, it’s fair to say that “web development” is not a clearly defined area. So much software today has some link or other to the web, it’s usually unfair to say definitively that something is or is not web development. So it’s a matter of degrees, and looking back on the software development I’ve been involved with I’d say the more web-oriented something has been the more I’ve had to deal with urgent issues, and the less web-oriented something has been the more I’ve had to deal with important issues. The most urgency comes when working on websites, and the least when working on applications which may or may not use the web for a bit of minor communication.

This issue of urgency also seems to be the perception and expectation of non-technical people. This is how we get the phrase “Internet time” and why for years I’ve heard people saying “but this is the Internet” as a justification for tight deadlines. This is something that continues today, even after the unsustainable boom and bust of the dot.com years.

When urgency is too pressing we risk doing a bad job, and releasing something which isn’t up to scratch. Of course, when urgency isn’t pressing enough we lose ground to our competitors. And of those two scenarios it’s the first one, doing a bad job, which is easiest to spot: a bug bites someone on the nose and you get to hear about it pretty fast. The second scenario, losing to the competiton, happens slowly over months or years, and by the time you slip into obscurity so many events have occurred it’s hard to pinpoint exactly where you went wrong.

Urgency is no bad thing, then, but I continue to be bothered by its dominance in web development. Too much urgency leads to a loss of quality and a reduction of strategic actions. Why does this happen in web development more than elsewhere? Here are some thoughts…

1. Updates are cheaper on the web

Rolling out a new version of a website onto a server shouldn’t be regarded as a trivial affair, but it’s a lot easier than other means of releasing software. So it’s easier to release a slightly rushed product if we know the only copies anywhere in the world live on servers which are entirely under our control.

Slightly more costly than updating our own servers is the job of the IT department which has to update an application on everyone’s desktop. Here, not only do you have to be sure the desktop machines are all switched on and ready to receive the software, but an error for one person could mean they cannot work at all the next day. More costly than that is the job of burning, boxing, and shipping a DVD to hundreds of retail stores.

For both those cases, and for many other ways of releasing software, the cost of repetition and the cost shipping something substandard is very high. But in many web applications, where the cost of distribution and the cost of error is relatively low, the consequences of problems that come from an urgent release are much less costly.

2. Problems hurt less on the web

As users, our relationship with websites and web applications tends to be less engaged than with most other applications. So although we all know undue urgency leads to more faults being released, it’s perhaps more forgivable on the web.

If I’m using a website and I encounter an error, I know the problem — the website — is theirs. I can come back later or try an alternative (there are always plenty of alternatives on the web). Either way, I’m probably not going to get very upset.

By contrast, if I encounter a problem with software running on my machine then any error is going to hurt me much more: that’s a problem with my software… it’s causing my machine to go wrong… I went to the trouble of installing that software and now it’s giving me problems… this is not good. Even if the software was installed by my IT department it’s still my software, and it’s core part of my daily work — finding a workaround is going to be difficult.

3. Competition is more visible on the web

One of the drivers of urgency is beating the competition, and if you and your competitors all work on the web, then they are likely to make (cheap) incremental releases, and you will notice almost every one.

If competitor A releases a new feature in week 1 then you’ll notice and ask how you can do that yourselves. If competitor B releases another new feature in week 2 then you’ll notice it and ask how you can beat that, too. By the time competitor C releases a third thing in week 3 you’ve got the pressure of the three things being delivered by your competition and nothing to show yourself.

Things are different as you move offline. Consider software companies in a competitve marketplace that see software ship features in batches. When your competition ships their new release with new features you’ll be able to assess all those features as a whole. Of course there will still be intense pressure to deliver, but your view will be more rounded and decisions of importance will play a greater role than decisions of urgency.

Consider also the internal team which delivers software which is internally-facing. Your life in this team will be no less demanding than in any other, but one thing you’ll be spared is much pressure from competitive teams. If you’re delivering the next version of the payroll system then there will be pressure of deadlines, there will be pressure of budget, but you’re very unlikely to get an unexpected new requirement due to the payroll software team at your competitor suddenly releasing their new e-mail alert module.

None of this is to say that competition is more fierce on the web. That may or may not be the case, but it’s not what I’m interested in here. The point is that the competition seems more fierce and is much more visible. This encourages urgency.

4. Boundaries are blurred on the web

A wordprocessor helps you write documents, a media player plays audio and video, but a website… a website doesn’t have clear boundaries.

Although I wrote above of competition, exactly who is a competitor on the web is not always certain. I once worked with an e-commerce music retailer who suddenly decided they needed to provide a web mail service for their customers. I’ve worked with a law firm who wanted to turn their brochureware site into a recruitment platform. Yahoo! evolved from a directory into a portal, embracing almost everything they could while the world struggled to understand what a portal might not be. Meanwhile, stepping away from the web, I don’t know of a spreadsheet which carries a discussion forum or an HR system which recommends movies.

While all this clearly accounts for the quantity of demands that occur in web development, it also accounts for the urgency because that steady drip-drip-drip of features that comes from our competition also comes from those who aren’t direct competition but are just out there on the web. The drip-drip-drip becomes a trickle (and maybe even a flood) pretty quickly, and the pressure to deliver new things is even greater.

5. It’s easier to have more stakeholders on the web

Because the web means so many things to so many people — contributing to those blurred boundaries — many more people within an organisation will consider themselves stakeholders. Marketing people see it as a marketing tool, the recruitment team see it as a recruitment tool, the sales team see it as a sales tool, and so on. And of course each one of them is right: it can be all these things.

With so many diverse stakeholders, we can expect so many more pressing deadlines, and hence much more urgency. More stakeholders also means communication and prioritisation is more difficult. It’s difficult to get everyone into the same room, it’s difficult to bring everyone to the same point of understanding, it’s difficult to weigh completely different kinds of requirements against each other. It doesn’t matter that everyone is working for the same company and therefore, at a higher level, the same goal. That goal is usually at too high a level to make a difference to everyone’s day-to-day needs.

The least pleasant example of this I remember is a colleague at a previous company who was the project manager for one of our clients. Although he was their single point of contact they managed to have several single points of contact, none of whom were very good at speaking to the others. He found he had responsibilities to the marketing manager, the distribution manager and the IT director. They all had pressing demands, and he had no contact with anyone who could bring them all into line. Indeed, that seemed to be by design, as we suspected the client used this as a way of squeezing us as a supplier. The relationship did not end happily. Although that’s a very extreme example it does highlight the kind of difficulties of having many stakeholders.

A final word

Cheaper updates, less damaging problems, more visible competition, blurred boundaries and more stakeholders. This tends to be the world of web development, and therefore a world in which urgency plays a more dominant role.

Yet we can think of web development projects in which importance dominates urgency. One which springs to mind is an online banking site: this is web development if ever anything is, but one in which doing things quickly will always be trumped by doing things properly (whatever “properly” may mean for that bank). But this is also an environment where many of the criteria I’ve listed don’t hold. Most obviously updates will be fairly costly, any problems will be very damaging, and the boundary of what the site does is very well defined.

So the criteria I’ve set out above are more to explain than define. Web development tends to take place in an environment that’s different from many other software development environments, and I think it’s those features of that environment that I’ve listed that explain why urgency predominates.

I said at the beginning this vexes me because urgency causes visible problems, and eliminating urgency would seem to help eliminate those problems. But I also said I’m usually dissatisfied with one unmet ideal or another. I’ve also worked in environments where there is a distinct lack of urgency. That, of course, leads to waste and stagnation. And that’s really, really, vexing…