Specs are useful for filling a communication gap. That gap may be one of team distance (the BA to the vast army of developers, too many to speak to all at once), or time distance (we’ll write this down now, because we may forget it in 3 months’ time when we start the work), or organisational distance (we’re going to give this to you, third party company, because this is precisely what we want you to build for us). There will be other examples.

But if those gaps don’t exist then you probably don’t need a spec. In particular, if you have four people in the company, and you don’t yet have any software, and are still validating your product idea, then a spec isn’t going to be useful. There, the best thing is to write the smallest thing you can, try it out, and repeat.

None of which is to say that all documents are bad. Far from it. It’s a good idea to write down a strategy (lest you forget it in the heat of the moment), an architecture (which you should aim for but may want to adjust), and some principles of various kinds (UX, engineering, product, etc).

But a spec is too often a document we could do without.

There are a few pertinent lessons:

1. Roll out slowly
2. Let systems co-exist
3. Always have a rollback plan

1. Roll out slowly

I’ve written before about the almost-inevitable pain of big bang launches, following Telegraph.co.uk’s update in 2007. Those lessons are true still. The only way to avoid that pain while still having a big bang launch is with a vast amount of additional resources (cost and time), which are rarely worth the value of what you’re gaining.

An approach with more likely success is to roll out piecemeal, replacing one user feature, or one system component, at a time. This is not sexy for engineers, who typically want greenfield projects and new technology, but nor is it sexy working 18 hour days to fix a torrent of bugs under a hail of angry user feedback.

This approach means breaking up the system into discrete components and replacing those. If the system doesn’t have such discrete components then the first job is to create them, going through the code and making logical fences. No this isn’t sexy, but let’s repeat: neither is working 18 hour days.

2. Let systems co-exist

It’s a computer science joke that every known software problem can be solved by adding an extra layer of indirection. However, sometimes that’s also a practical solution. This allows you to build an old and a new system side by side, and funnel specific users/activity/processes/etc to either one. As you build up the capabilities of the new system you can funnel more and more activity to it.

Two architecture cast studies:

In the first (Architecture 1, above) we needed to replace a proprietary application server with one over which we had much more control. Critically, the new application server needed to be able to execute the same scripts, of which there were thousands.

Our first-release replacement app server had only the most basic capability, and so could only execute a tiny fraction of the scripts. So we put in a layer of indirection which filtered requests to either the old or the new server. As we enhanced the capability of the of new application server we filtered more and more requests to it, until we were finally able to switch off the old one.

In the second (Architecture 2) we needed to migrate our user data to a new, more flexible database (MongoDB), despite some of the application code being rather old and tied closely to the legacy database. The solution was to introduce an internal API, ensure all code used that API instead of direct database access, and then add the new database under that. The two systems co-existed for a long time as their capabilities were brought together and tested for consistency.

3. Always have a rollback plan

Once we have a step-by-step approach to moving from one system to another we have to recognise that some steps may not work flawlessly. In this case we have to be able to switch back, because fixing a problem in situ may take too long. This is something that can be built into the indirection layer’s config system, or into the release mechanism (allowing easy rereleaes of the previous version), or be switchable in the application’s own config  (see Flickr’s approach as one example).

It’s still difficult

None of this is easy, of course, but it’s the kind of thing were the effort is in the up-front planning and the initial (possibly time-consuming) prep to tidy up the old system, all for the sake of a smoother rollout. Launches are exciting, but they can be exciting in the intended way at the same time as going boringly to plan.

Meanwhile the folks at Delicious are slowly restoring many lost features, and I wish them well, because we’ll all win if they do.

These are questions that arose most recently in a post by Rob Bowley. It’s well worth a read, with lots of good references, and there’s a terrific discussion underneath. My own answer to these questions is that any estimate can be useless or misused (even those of an hour), and any estimate must be placed in a context to be meaningful.

Estimation considered harmful

Rob makes a lot of good points. Here are some I agree with:

• Misuse of estimates is the root of many, if not most, failed software projects being considered a failure.
• Almost all of us are really bad at estimating in these situations.
• For most organisations the definition of tech success is “on time and on budget”
• When asked for long range estimates it’s irresponsible to just say No and walk away.

And here’s what I disagree with:

…we should start being brave and just saying no to requests to provide estimates on anything more than 1 or 2 months (at most) work. [...] What’s the least worst situation to be in? Having an uncomfortable situation with your boss now or when – based on your estimates – the project’s half a million over budget and 6 months late? [...] My advice to anyone who’s put in the situation of providing long range estimates is to be brave and simply say it’s not something we can do, and then do your best to explain why.

Being meaningful

Just saying No isn’t helpful — even with an explanation. The worst that can happen isn’t bringing forward an inevitably-uncomfortable situation with your boss. The worst that can (reasonably) happen is that your boss systematically turns to others who are more on his or her wavelength, and you get seen as a fist-waving problem employee who is consistently sidelined.

To me, the truth is in this Twitter exchange I had with Rob, and Rachel Davies, while he was writing the piece:

[View the story "Estimation problems, with Rob Bowley" on Storify]

The key text is when Rob says that discussing project duration is perfectly valid only if you can say anything meaningful about it. What’s meaningful varies. Even saying something will take one hour may not be sufficiently meaningful in some situations:

Chief: Can you write a script that shuts down all the launch servers?

Technician: Sure. I estimate that will take an hour. [Thinks: I'll start it after lunch.]

Chief: Thank goodness. Because apparently someone’s hacked them, and they’ve been reconfigured to launch a nuclear strike against Italy in [looks at watch] 1 hour and 2 minutes. But I’m reassured now we’ll be okay.

If that 1 hour estimate is out by just 3 minutes then there are very bad consequences. Suddenly it looks as though the estimate didn’t provide the right meaning to the chief.

In the comments on Rob’s piece, Paul Dyson gives two detailed examples of where long range estimates were presented in a meaningful manner. Each was a situation where figures were not presented as standalone numbers, but part of a larger response that made up a business solution: we believe this investment will achieve this outcome, and we will track success continually in a dialogue with you.

Presenting standalone numbers in response to specific questions divorced from the larger context is responding to the wrong question. Having a meaningful dialogue with your peers is more productive.

In my time at the Guardian we built our micro-apps framework with specific intention to support a specific project: our discussion platform. There was a time and budgetary estimate, but the end goal of delivering the discussion platform was what counted.

Guardian in America has a specific budget, too. It launched recently with a new home page focused on a US audience. That’s far from all that will be delivered, but you can be sure the project’s early goals are being met with that early delivery, and there is a feedback loop which comes from using this for real. One day the budget will be used up, but before then the priority of individual features will have changed in the light of real-world experience. That in turn means the priority of the project will be on delivering business objectives, not on sticking to a narrow view of on-time-on-budget.

Making estimates part of the bigger picture, to give them appropriate meaning, is what creates success.

In my previous example our project had a revenue projection of £150k to £500k. But suppose the project was expected to cost £200k. Then that range encompasses both profit and loss. What can we do?

Here are some suggestions:

1. Collect more data

We may be very confident of our potential revenue, but perhaps we could be more confident. More research data should change our perception of what might occur, and so narrow the range of expectations. This might include customer research or research into other companies’ experiences.

Of course our findings may just serve to confuse the issue: we may collect six examples of other companies’ experiences only to find they differ wildly. In such cases we should be more discerning about how we see the data. For example it would be a good idea to learn more about each example and see which are closest to our own proposal.

2. Break down the problem

Our revenue projections (or other measurement) are likely to be a factor of several other elements. If we can break down the problem we can perhaps analyse those elements more easily. With more confidence over simpler measures we ought to be able to make tighter estimations of the whole.

For example, revenue may be seen as a factor of market size, typical customer spend, product visibility and ability of the product to meet customers’ needs. There may be market data readily available on market size and customer spend; we should be in a position to gauge product visibility if that’s a result of our own marketing; and focus groups or similar may be able to help us understand how the product is seen to meet customers’ needs.

3. Move the breakeven point

If we’re worried that we may generate £150k to £500k from a product or project that costs us £200k then maybe we should seek to reduce the costs. Of course if we reduce functionality or quality then there is likely to be some detrimental effect to the revenue. But we may be able to find high cost/low value items which change the game sufficiently.

4. Change the factors that influence the range

It may be that we can reduce uncertainty by actually influencing some of the critical elements. If it transpires that customers don’t see the product as meeting their needs sufficiently then we should change that. That may be by changing the product, or by improving our marketing.

Or we may find it’s useful to do the opposite of our previous suggestion of reducing cost: increasing quality (and adding a bit of cost) may pay extra dividends by increasing take-up. This kind of thing can be seen in places like the Apple App Store. I am at least twice as likely (probably four to five times more likely) to buy an app rated 4.5 stars than if it was 3.5 stars. Yet the difference in production cost to get to that level would have been far less than two-fold.

I’m sure there are more ways to respond to an insufficiently compelling range of possibilities in a project proposal. Overall, though, all these actions have the same beneficial effect I pointed to before: the relevant issues are surfaced so that they can be addressed appropriately and we increase the likelihood of our project’s success. And they are much more likely to be surfaced an addressed if we also surface that fact there is a range of outcomes, rather than project a single outcome.

A typical proposal (I’m thinking mainly of internal projects) will make a claim like “This project will add 500,000 unique users in the first year” or “This will bring in £400,000 of new revenue by October” or “This will save £250,000 next financial year”. I know because I’ve written some of them. In choosing what number to use you don’t want to put in anything too low or the project won’t look worthwhile, so you tend to put in something that’s a little generous (but not implausibly so) and assume that all the contributing factors will work out for the best — the marketing, the timing, the user take-up, and so on. Depending on the audience you’re presenting it to you might even have some evidence for your projection.

The problem is, of course, that it won’t happen like that. Some things will work out well, some not so well. Everyone knows that, and if you’re facing a sceptical inquisitor then you’re just setting yourself up with less credibility before you start. Thinking back on the last such proposal I wrote I can imagine a conversation with a fantasy inquisitor that might have gone like this…

Fantasy inquisitor: You say this will bring in £400,000 before October.

Me: That’s right.

FI: Can you guarantee that? Can you guarantee £400,000 of new revenue?

Me: [fearing liability, and wondering which members of my family might get taken away if it doesn't happen]: No, of course I can’t absolutely guarantee it.

FI: So what are you saying about the £400,000 if you can’t guarantee it?

What indeed? The truth is that will be the revenue if things generally go well. We might make even more. But realistically we might make less. Being honest with ourselves and our audience will help everyone. Let’s return to the scene…

FI: [repeating for the sake of the drama] So what are you saying about the £400,000 if you can’t guarantee it?

Me: I’m confident we’ll make that.

FI: How confident?

Me: Really quite confident.

FI: So you’re “really quite” confident it will make at least £400,000. And I suppose it could make us as much as… what? £800,000?

Me: Er, no I don’t think it will go that well. Maybe £500,000.

FI: So you’re confident it could make as much as £500,000. And you can be confident it won’t generate a penny less than £400,000, can you?

Me: Well, it certainly could make less, I suppose, if not everything went to plan.

FI: Would you be surprised if it didn’t make us anything at all?

Me: Yes, I would. Even being realistically pessimistic it really should make at least £150,000.

FI: Okay, so you’re very confident we’re looking at making £150,000 to £500,000 with this?

Me: Yes, I’m very confident of £150,000 to £500,000 of new revenue.

So our single (and therefore implausible-sounding) figure becomes much more realistic when we change it into a range, even if it goes down at one end. It also goes up at the other end.

I think this goes a long way in dealing with a sceptical audience becaues it is more realistic. For the same reason we should be more confident when we talk about it, too.

I’ll add two final notes. First, there’s an important question that needs to be asked, which I’ve omitted: What is your evidence for this? This may be calculations based on observations, historical data from similar projects in the past, market research, etc. Either way, it’s important to know that the figure(s) in question aren’t entirely random.

Second, the statisticians in the audience will point out that “very confident” is almost meaningless when we could say “90% confident” or attach some other actual number. I certainly appreciate that, and mostly concur. But for me a significant, easy, and meaningful step forward is to move from single figures to more realistic ranges. And this should promote more honest and confident dialogue.

The Eigenharp and the Open Platform

The Eigenharp is a new kind of device. It’s a musical instrument with several patents already filed, and incorporates 132 keys with three-colour LEDs and a breath pipe. The keys operate in three dimensions (so you can exploit, say, pressure, pitch bend and filter effects with one finger) and are sensitive to within one micron. So while it’s compatible with MIDI, that’s an incredibly out-of-date protocol compared to what you could do with it.

The Open Platform has been well-documented here and elsewhere: a full content API directly into the Guardian’s content database, plus a store of raw data on which our journalists base some of their stories.

Although they are clearly two different kinds of beasts a remarkable similarity struck me in that they are both not just new products, but really new products. There is no clearly defined place for them in either of their markets, and no predictable success path for either one. The success of each is dependent entirely on other people’s innovation. The Eigenharp’s success is dependent on musicians taking their creativity into new directions — no-one to date has written music for an Eigenharp. The Open Platform’s success is dependent on developers doing innovative things with the new data they have access to.

So how do you launch a product that’s really new? The similar steps I can see are…

1. The controlled buzz

Eigenlabs (the creators of the Eigenharp) and the Guardian (creators of the Open Platform) managed to seed a few select people with their product — musicians and developers respectively — and some small details leaked out. This clearly did a couple of things.

First, it helps you get the product right. I know from my conversations around the Open Platform that what might seem a good idea to one or two people close to the product can provoke strong negative reactions when you float it past an informed outsider. When you’re inside the company you can easily think too much and lose the big picture, so it’s good to get an informed but independent view before you go public.

Second,  creating a buzz protects you from a bit of unforeseen negative reaction — if people are excited about your product and want it to succeed they will forgive some minor mistakes. You can see Eigenlabs achieved this with this low-fi video of two guys playing the James Bond/Moby theme on YouTube. Eigenlabs didn’t create this themselves, but they let “close friends” do it, and you can see the reaction in the comments underneath: “What the hell!! This is awesome”, “where can I buy one of these?”, “dear god this is epic” and so on.

The benefits of this are well explained by Lance Ulanoff at PCMag.com. Here he compares the lacklustre launch of the Motorola Cliq with the hype from the January 2009 announcement (six months before the launch) of the Palm Pre:

Speaking of Palm, it has a good bit in common with Motorola right now. The Pre (and now Pixi) is its hail-Mary pass. If the new phone and WebOS platform fail, Palm will be done. Back in January, however, Palm ran the best product rollout event I have ever attended. It perfectly conveyed the company’s excitement and all that is good about the Pre. I remember tweeting the event with mounting excitement. Today, I tweeted the Motorola event with mounting confusion.

What Palm did in January was give the Pre a good hard shove off the shore. Those waves of excitement produced a good six months of positive press. Eventually, Palm and the Pre hit some rough waters—a shipping delay, the slow delivery of the SDK and tiny list of apps that has yet to grow. Still, that first day set the tone.

Then after the buzz is…

2. Openness at launch

There’s much that doesn’t need to be said about the launch event, but one thing that struck me about both the launch of the Eigenharp and the launch of the Open Platform was a common ethos: the honesty and openness of the presenters, and the openness of the products.

At both events the “real users” (musicians/developers) were on-stage demonstrating their early adoption of the product and open to questions from the audience. These are the kinds of people who most marketing and PR people would lock in cupboards at a press event for fear they arrive without the regulation press-on smile and go off-message. But when a product’s success relies on innovation and creativity from its real users then it’s important that intended audience hears an authentic voice.

Also, there is an openness about both products. Aside from the very concept of opening up content and data, the Open Platform gives a lot of latitude to developers, including (unusually) commercial use. Meanwhile Eigenlabs are open-sourcing their software. Again, because each product’s success relies on people taking it in unexpected directions it’s important for there to be as many opportunities as possible for that to happen. Otherwise its success is stifled.

And then finally there’s…

3. The follow-through

Well, it’s early days for both products. The Open Platform API is still in beta, but you can see some developments now, such as the launch last week of the Applications Gallery. It’s even earlier for the Eigenharp. Company founder John Lambert was asked what well-known musicians he’d like to see using his creation and he said that a number of high profile people do have the instrument but that he cannot name any names yet. Clearly some really interesting things are going to happen some time soon.

If a product really does depend on the creativity of others then you can’t — and mustn’t — be too controlling of what those people do with it. (When I spoke to him John seemed slightly apprehensive about the quality of some of the things people might produce around the Eigenharp; I bet Matt McAlister didn’t anticipate a swearword tracker coming out of the Open Platform.) But if you can’t control what others do, you can at least show some examples of what might be achieved, and that’s what the follow-through is doing with each of these products.

1-2-3, easy as A-B-C

Now if any marketing people have stumbled across this blog post I wouldn’t be surprised if they were horrified by the naivety of these observations — all of this might just be first grade A-B-C of marketing for them. But I find this fascinating. As a tech person it’s very easy for me to focus on my own role and not spend much time wondering what challenges are faced by my colleagues in other departments. Here are two products whose paths to success are unusually dependent on the unknown and whose stories will be well worth watching. By looking at the communication that’s gone on around them the significance of other people’s role becomes much more apparent. Technology success is about much more than successful technology.

Many people who read just a little about Agile development think there are no fixed commitments. It’s true that there is constant reprioritisation of work, but that generally operates at the task level, and there is still a need to set goals and stick to them. After all, how else can you give the people who sign the cheques reassurance that you’ll deliver what they want?

So when we planned R2 we also set milestones marking when we would make the major launches of each section. Each milestone was stated very simply at the high level: launch the homepage, launch the Environment section, etc. These were the targets we put before our senior stakeholders, and they were what we were accountable for.

At a lower level we planned for each launch to include specific features: the Culture launch would include a special component to list the critics; the Sports launch would include a component showing the latest betting odds from Blue Square; and so on. It was these lower level details which were always up for reprioritisation. As long as we could deliver sufficient features to allow the launch of each section we could fairly be said to be hitting that milestone and satisfying our senior stakeholders.

There’s an introduction and discussion on the more public-facing aspects of this, kicked off by Emily Bell. For my part, I want to talk briefly about one of the most remarkable behind-the-scenes aspects of it: how we got the weekend launch to go so incredibly smoothly.

The secret is that the weekend’s work was only the final step after a great many before it, all of which were safely out of the way before the weekend…

1. Software release

The actual software was released some weeks ago, in early January. This means that by the time of the launch it had been in use for some time, almost all the lines of code having been executed several hundred (and in some cases thousand) times already, in the production environment.

Even that January release was only an enhancement of previous releases which have been going out fairly quietly over the previous few months. The latest one included new internal tools, and updates to tools, to support some of the new features that are visible today.

2. Building the pages

Meanwhile editors, subeditors and production staff have had to learn to use those tools. They’ve been using them to migrate a lot of the older content into the new-format pages. You might think that could be done by machine, but that’s not the case. Since we’re also changing our information architecture — adding a lot of semantic structure where previously there was only presentational information — it takes an informed human being to know how to convert an older page into its newer format. A real person is needed to understand what a piece of content is really about, and what it isn’t about but merely mentions in passing. We also need people to look at the new keyword pages (for example, the one on immigration and asylum, or the page on Kosovo), which are populated mostly automatically, and for them to highlight the most important content: the most important content won’t necessarily be the newest.

This work had been going on for many weeks before the weekend launch. The January software release brought in some tools refinements to help them tidy up final loose ends (and no doubt some more tidying will happen over the next couple of weeks).

You can see from this that it’s about much more than mere software. The software enables people to do the editorial work, and the editorial work has been going on for some considerable time. Everything they’ve done has also been tested and previewed, which allows them to see what our external users would see were it to go live. Again, this exercises the software fully, but only within the internal network, before it’s exposed to the outside world.

3. Rehearsals

The work for the weekend launch is mainly running a lot of database scripts to make various new URLs live and decommission old ones. The reason this is such a huge launch is that there’s over ten years’ worth of news content to expose, as well as new functionality and designs.

We couldn’t trust the scripts to work first time (of course), so we spent a lot of time copying the production content into a safe environment, and rehearsing the process there, with real data. We needed to be sure not just it would work, but also how long it would take (considering any hardware differences), and change the scripts and process accordingly.

4. Launch

Finally, after all the rehearsals, the real deal. The work to run the database scripts and raise the curtain on various features ran over Saturday and Sunday, but it was calm enough and organised enough that the team needed to work only slightly extended working days.

So the big launch was a culmination of a huge amount of effort, and the weekend work was after an awful lot of practice. There were a couple of sticky moments, but nothing the team couldn’t recover from in a few minutes. As one of the developers remarked towards the end of Sunday: “The fact that today’s been really tedious is a good thing.”

What we can see now includes

• New navigation at the top of most pages;
• UK and world news sections;
• A new layout for the Observer homepage;
• A new politics site;
• An audio homepage…
• …with each piece of audio content identified as an individual resource with its own URL;
• An A-Z index of subjects…
• …and contributors;
• More meaningful related links down the side of articles;
• An “article history” link for every article in the new format, in the name of transparency;
• Page-by-page and section-by-section navigation around the latest edition of the Guardian and Observer;
• Lots of new layouts for editors to choose from, like this and this and this;
• And a new favicon.

What’s next…

We’ll be ironing out a few things over the next few days, but everything’s gone to plan for now. And then, as Emily says, there’s still sport, arts, life and style, and education to do.

However, not everyone at the website will have been downing champagne as the moment the new site went live, because in the first few hours it was clearly having some performance issues. We’ve all had those problems some time in our careers (it’s what employers call “experience”), and it’s never pleasant. As I write it looks as though the Times Online might be getting back to normal, and no doubt by the time you read this the problems will all be ancient history. So while we give their techies some breathing space to get on with their jobs, here are three reasons why making performant software is not as easy as non-techies might think…

1. Software design

A lot of scalability solutions just have to be built in from the start. But you can’t do that unless you know what the bottlenecks are going to be. And you won’t know what the bottlenecks are going to be until you’ve built the software and tested it. So the best you can do from the start is make some good judgements and decide how you’d like to scale up the system if needed.

Broadly speaking you can talk of “horizontal scaling” and “vertical scaling”. The latter is when you can scale by keeping the same boxes but beefing them up — giving them more memory/CPU/etc. The former is where you can scale by keeping the same boxes end-to-end, but add more alongside them. Applications are usually designed for one or the other (deliberately or not) and it’s good to know which before you start.

Vertical scaling seems like an obvious solution generally, but if you’re at the limit of what your hardware will allow then you’re at the limit of your scalability. Meanwhile a lot has been made of Google’s MapReduce algorithm which explicitly allowed parallelisation for the places it was applied — it allowed horizontal scaling, adding more machines. That’s very smart, but they’ll have needed to apply that up-front — retrofitting it would be very difficult.

You can also talk about scaling on a more logical level. For example, sometimes an application would do well to split into two distinct parts (keeping its data store separate from its logic, say) but if that was never considered when the application was built then it will be too late once the thing has been build — there will be too many inter-dependencies to untangle.

That can even happen on a smaller scale. It’s a cliche that every programming problem can be solved with one more level of indirection, but you can’t build in arbitrary levels of indirection at every available point “just in case”. At Guardian Unlimited we make good use of the Spring framework and its system of Inversion of Control. It gives us more flexibility over our application layering, and one of our developers recently demonstrated to me a very elegant solution to one particular scaling problem using minimally-invasive code precisely because we’d adopted that IoC strategy — effectively another level of indirection. Unfortunately we can’t expect such opportunities every time.

2. Devising the tests

Before performance testing, you’ve got to know what you’re actually testing. Saying “the performance of site” is too broad. There’s likely to be a world of difference between:

• Testing the code that pulls an article out of the database;
• Testing the same code for 10,000 different articles in two minutes;
• Testing 10,000 requests to the application server;
• Testing 10,000 requests to the application server via the web server;
• Testing the delivery of a page which includes many inter-linked components.

Even testing one thing is not enough. It’s no good testing the front page of the website and then testing an article page, because in reality requests come simultaneously to the front page and many article pages. It’s all very well testing whether they can work smoothly alone — it’s whether they work together in practice that counts. This is integration testing. And in practice many, many combinations of things happen together in an integrated system. You’ve got to make a call on what will give the best indicators in the time available.

Let me give two examples of integration testing from Guardian Unlimited. Kevin Gould’s article on eating in Barcelona is very easy to extract from the database — ask for the ID and get the content. But have a look down the side of the page and you’ll see a little slot that shows the tags associated with the article. In this case it’s about budget travel, Barcelona, and so on. That information is relatively expensive to generate. It involves cross referencing data about the article with data about our tags. So testing the article is fine, but only if we test it with the tags (and all the other things on the page) will we get half an idea about the performance in practice.

A second example takes us further in this direction. Sometimes you’ve got to test different operations together. When we were testing one version of the page generation sub-system internally we discovered that it slowed down considerably when journalists tried to launch their content. There was an interaction between reading from the database, updating the cache, and changing the content within the database. This problem was caught and resolved before anything went live, but we wouldn’t have found that if we hadn’t spent time dry-running the system with real people doing real work, and allowing time for corrections.

3. Scaling down the production environment

Once you’ve devised the tests, you’ll want to run them. Since performance testing is all about having adequate resources (CPU, memory, bandwidth, etc) then you really should run your tests in an exact replica of the production environment, because that’s the only environment which can show you how those resources work together. However, this is obviously going to be very expensive, and for all but the most cash-rich of organisations prohibitively so.

So you’ll want to make a scaled down version of the production environment. But that has its problems. Suppose your production environment has four web servers with two CPUs and six application servers with 2GB of RAM each. What’s the best way to cut that down? Cutting it down by a half might be okay, but if that’s still too costly then cutting it further is tricky. One and half application servers? Two application servers with different amounts of RAM or CPU?

None of these options will be a real “system in miniature”, so you’re going to have to compromise somewhere. It’s a very difficult game to play, and a lot of the time you’ve got to play to probabilities and judgement calls.

And that’s only part of it

So next time you fume at one of your favourite websites going slow, by all means delve deep into your dictionary of expletives, but do also bear in mind that producing a performant system is not as easy as you might think. Not only does it encompass all the judgement calls and hard thinking above (and usually judgement calls under pressure), but it also includes a whole lot of really low-level knowledge both from software developers and systems administrators. And then, finally, be thankful you’re not the one who has to fix it. To those people we are truly grateful.

• Individual productivity metrics don’t work.
• Productivity-like metrics are still useful…
• …but they don’t tell the whole story

Individual productivity metrics don’t work

Joel O’S states that if you incentivise people by any system, there are always opportunities to game that system. My own experience here is in a previous company where we incentivised developers by how much client-billable time they clocked up. Unfortunately it meant that the developers flatly refused to do any work on our internal systems. We developed internal account codes to deal with that, but it just meant that our incentivisation scheme was broken as a result. Joel has other examples, and Martin Fowler discusses the topic similarly.

Productivity-like metrics are still useful…

Agile development people measure something called “velocity”. It measures the amount of work delivered in an iteration, and as such might be called a measurement of productivity. But there are a couple of crucial differences to measuring things such as lines of code, or function points:

• Velocity is a measurement of the team, not an individual.
• It’s used for future capacity planning, not rewarding past progress.

Velocity can also be used in combination with looking at future deadlines to produce burndown charts and so allow you to make tactical adjustments accordingly. Furthermore, a dip in any of these numbers can highlight that something’s going a bit wrong and some action needs to be taken. But that tells you something about the process, not the individuals.

The kick-off point for Joel’s most recent essay on the subject is a buzzword-ridden (and just clunkily-worded) cold-call e-mail from a consultant:

Our team is conducting a benchmarking effort to gather an outside-in view on development performance metrics and best practice approaches to issues of process and organization from companies involved in a variety of software development (and systems integration).

It’s a trifle unfair to criticise the consultant for looking at performance metrics, but one has to be careful about what they’re going to be used for.

…but they don’t tell the whole story

A confession. We track one particular metric here in the development team at Guardian Unlimited. And a few days ago we recorded our worst ever figure for this metric since we started measuring it. You could say we had our least productive month ever. You could. But were my management peers in GU unhappy? Was there retribution? No, there was not. In fact there was much popping of champagne corks here, because we all understand that progress isn’t measured by single numbers alone. The celebrations were due to the fact that, with great effort from writers, subs, commercial staff, sponsors, strategists and other technologists we had just launched

• a refreshed and reorganised Arts site,
• a brand new Arts blog,
• our first ever dedicated Music site
• with its own Music blog,
• and an exclusive video,
• and we still made time to help our colleagues over at Guardian Weekly launch the new Guardian Abroad site.

A bad month then? Not by a long shot. The numbers do tell us something. They tell me there was a lot of unexpected last-minute running around, and I’ve no doubt we can do things better the next time. It’s something I’ve got to address. But let’s not flog ourselves too much over it — success is about more than just numbers.