Big lessons from a little project

I’ve just finished a fortnight’s holiday which I (foolishly) spent mostly in front of a PC developing a never-ending little application. But unexpectedly, despite the trivial nature of my project, I rediscovered a number of important lessons more usually associated with serious application development.

The software I’m writing is a just a little Firefox plugin. I’ve been fiddling with it for so long we’re almost onto the second major release of Firefox since I started, and yet it’s probably not much more than a couple of hundred lines long. You can see it really is a minor enterprise. Despite this it’s been quite a surprise — quite a shock, even — to be reminded of some industrial-strength truths in a small and personal environment.

And they are…

1. Damn, writing software is difficult

What we non-developers know about software we know either by observing or by talking to those who do it. But anyone who wants to be trusted won’t complain about their job or bore you with details they know you don’t want to hear, and so you’ll never hear about everything there is to know about software development, even when you ask.

So one thing I discovered — again — is that writing software is really difficult. Sometimes I was flying, but more often I was crawling: piecing together information from different sources, trying to understand what was possible, learning all kinds of technologies (Javascript, XUL, XPCOM), trying use them well, but more often trying to get them to work at all.

It’s made me respect all the more the people I work with every day who make it look so easy. Every line you write is pure logic and needs to work 100%. This is not like writing a letter or a sales proposal, where a sentence that’s only 95% perfect is more than adequate. This is like writing a legal contract from scratch for a particularly unpredictable world. Every line needs to be watertight.

Aspects of commercial software development2. Simple design really is difficult

Simple design is another way of saying that the software should be easy to work with and modify. You’d think this would be hard to get wrong, particularly with something which is being built as new: surely you just add small, simple pieces one at a time, with each one adding the next feature on the list. What could go wrong?

I found out first hand that simple design is difficult to achieve, even in very simple scenarios. For example, my application has a class which handles the user’s preferences. It needs to be initialised for when the application starts, and it needs to take and save the preferences when requested. All this takes place within a single object. So I wrote an initialisation function which set up the initial preferences according to how it’s been configured, I tested it, and all was well. Then I added some functions to take updated values and save them, tested those, and all continued to be well. Total: about 20 lines of code over four functions.

Then I ran the whole in an integrated environment and weird errors started occurring. It took me about two hours to figure out what was going on: it was a strange combination of unexpected start-up values, Firefox calling the initialisation function at unexpected times, and some confusing logic of my own which was supposed to protect double-initialisation. In the end I decided the best solution was to throw out the whole idea of initialisation. Now the object just takes and save preferences, and you can do that when the application starts if you want. Total: 5 lines.

The point of this is that an apparently simple and obvious design was actually too complicated to sustain. I’m very pleased that the solution was to delete lines and simplify. But I could only do this so easily because I had complete control of the code. In other circumstances there might have been other systems which were relying on that initialisation code (however flawed it may have been), and I might have had to add to the existing complications to solve my problem; or I might not have had the time to take a fresh look at the code and simply built around the flaws out of a sense of fear of touching the wrong thing.

This is a tiny example from a tiny piece of work, but it showed me how easy it is to go wrong with a design, and how easy it is to produce software that is complicated, hard to understand, and time-consuming to fix and evolve.

3. Learning a language is more about culture than syntax

I sometimes get CVs from people who claim to know about 10 programming languages, and I’m always doubtful. Just because you’ve written an application in a language it does not follow you’ve done it well. Knowing the syntax is only the first step. You also need to have good knowledge of any libraries, and finally you need to know how to work with the grain of a language. This means you’ll use different idioms, and structure your solutions in different ways.

In my own case I’ve been writing Javascript, but it stinks because I’ve tried to use it like Java. I’ve been stuck in my old Java ways, like creating classes and carving out a deeply-nested namespace. It’s Javascript, Jim, but not as we know it. It works, it makes sense, but it looks clunky and… well, it just feels wrong, dammit.

Javascript is a prototyping language. I can even tell you what that means, but only with my head, not with my heart. I use the prototyping as a hoop to jump through to get it to do the Java-y things that I know I shouldn’t be doing in the first place. Being a prototyping language doesn’t mean Javascript is a second-class language, or a dumbed-down Java. It means it’s a different kind of language to Java and should be treated as a first class language with its own ways of getting things done.

It’s a cultural thing, and you can’t claim to really know the language if you don’t operate comfortably in its culture. I don’t really know Javascript.

4. How did I ever live without automated tests?

Possibly by not spending my holidays sat in front of a PC. But aside from that, I continue to wonder at the marvel that is automated testing, and unit testing in particular. To be able to implement a change and not have to trouble your brain about the consequences is very liberating, allowing you to move ahead with confidence. It does take some work to set up the environment, but the results are worth it.

That said…

5. Your automated tests won’t cover everything

In one of my functions I unexpectedly found a truck-sized hole which had gone undiscovered despite seemingly comprehensive automated test coverage. (A loop which had a “break” when it should have had a “continue”, meaning great swathes of actions got skipped in most circumstances.) I only discovered this through integration testing (which is the fancy name I’m using for what was really “trying it out”), and found that a quirk in my unit testing setup had caused the mistake to be missed. Once I had found the cause I adjusted the main code to be more predictable and put in an automated test to trap the error, but it was only discovered through real hands-on testing.

6. A strong IDE sustains motivation as much as anything

Although I was using Eclipse for development, when working with Javascript it really doesn’t offer the comprehensive support you get with Java. Because Javascript is a dynamic language, and no doubt also because of the state of JSEclipse, there’s very weak support for code completion, refactoring and so on.

The consequence of this was a loss of code-writing speed, but much more than that I was suddenly able to see how easily a weak IDE allows bad software to be produced. There were many occasions when I knew that I should tidy up or refactor something, but was then suddenly hit with a premonition of the tedious steps I’d have to go through: working out which files to pick on, the manual search-and-replace, checking the context before I made a change. I had to force myself to get on with the tidying up despite knowing how painful it would be, focusing on the long-term results, and safe in the knowledge that for this little personal project I didn’t have a deadline.

It became clear to me that so many people must come under a constant barrage of pressure, with only their current strength of character to defend against the pressures of deadlines and short-term wins. It’s inevitable that too often they will give in to those pressures, leaving cumbersome code building up, and ultimately gumming up the works of the system. A strong IDE removes the barriers to those virtuous tasks of improving design and allows you not only to do your job, but to do it well.

7. Estimation is difficult

Which is just an excuse for the shameful truth: all my estimations were out by a factor of four. This is embarrassing because it’s not as if I’ve never written software before.

In retrospect the mistake I was making was to look at the component parts of a task, guess how long they’d each take, and add them up. What I should have done was try to take the forthcoming work, identify similar previous tasks, and from past experience see how long they actually took.

At the end of the day experience is good, but it’s how you use it that really counts.

8. You can’t know all your requirements up front

This is familiar to anyone who’s bought into Agile, but it hits home hard when it’s you who’s the user setting the requirements.

I’ve written countless requirements specifications in the past (in 60-page documents, on task cards, wherever) — I thought I really ought to be clear-thinking enough to know my own requirements up front. Wrong again. As the UI came together, as one idea sparked off another, and as I had chances to step away from the code to think about things from a distance, I started to see that my feature set was really rather disjoint — almost random. These were moments of clarity that on the one hand caused me to add requirements, but in doing so I was recasting the software with a new perspective. I was starting to see what the software should be doing, which was not quite the same as what I’d started out on.

Release date

Fortunately I’ve not sent out a press release announcing a release date, held a press conference, or hired the London Eye for a glitzy media event. I’m just writing software for fun, and at this rate it’s probably not going to ever see the light of day. But even then, it’s been startling to find that the germs of some of the Big Ideas of software development are still present in the smallest of projects.

QCon London 2008: A Michelin-starred deli

There were very few moments for me during QCon London 2008 of earth-shaking enlightenment — if any. But every hour of the three days of the conference there were insights and guidance that could be tucked away, and reused later to save hours, days or weeks of time elsewhere. Snake-oil salesmen where thin on the ground, and instead there were dozens of people saying one or both of:

  • This is what we did; and
  • This is what you can do.

No magic, no silver bullets, but plenty of solid advice and experience.

A good example of both of these was Randy Shoup of eBay. He had nothing to sell (other than the good name of eBay, perhaps) and his presentation was very clearly constructed to show their principles of scalability, and some concrete examples of how these work in practice. You probably wouldn’t use their periodic batch processing method to generate recommendations — if only because it’s odds on you don’t have a recommendation system — but you could take the overarching principle of “async everywhere” and apply that to the next scalable application that you need to work on.

Even the very specific presentations contained valuable points that could be generalised and reused. For example, Matt Youill and Asher Glynn of Betfair talked through how they scaled the transaction processing on their servers by a hundred-fold. Guardian.co.uk doesn’t need that kind of throughput, so the details were primarily of intellectual benefit. But a key practical lesson was how they approached the problem: by presenting it to industry players as a challenge carrying great kudos to the winning company.

All of this was summed up very nicely by the team from BBC News: John O’Donovan, Kevin Hinde and Ross Heritage. They were asked how they managed performance testing for the iPlayer. John spent a few moments describing some of the techniques they employed, but got to the point when he realised the audience really wanted some eye-opening enlightenment which he didn’t have. At this moment Kevin stepped forward and said straight out “There’s no secret sauce”. Indeed not: they just work hard and stick to strong principles.

QCon offered little in the way of secret sauces, but it did contain dozens and dozens of great ingredients you could take away and use to concoct your own wonderful dishes.

And with that analogy pushed to breaking point, I think we should leave it there.

Guardian.co.uk: Steps to a smooth launch

At the weekend the Guardian website went through one of the most significant transformations in its history: we moved our news, politics and Observer content into the new design and new content management system, and we simultaneously launched a lot of new functionality, both internal and external.

There’s an introduction and discussion on the more public-facing aspects of this, kicked off by Emily Bell. For my part, I want to talk briefly about one of the most remarkable behind-the-scenes aspects of it: how we got the weekend launch to go so incredibly smoothly.

The secret is that the weekend’s work was only the final step after a great many before it, all of which were safely out of the way before the weekend…

Guardian.co.uk global navigation bar1. Software release

The actual software was released some weeks ago, in early January. This means that by the time of the launch it had been in use for some time, almost all the lines of code having been executed several hundred (and in some cases thousand) times already, in the production environment.

Even that January release was only an enhancement of previous releases which have been going out fairly quietly over the previous few months. The latest one included new internal tools, and updates to tools, to support some of the new features that are visible today.

Guardian and Observer archive navigation2. Building the pages

Meanwhile editors, subeditors and production staff have had to learn to use those tools. They’ve been using them to migrate a lot of the older content into the new-format pages. You might think that could be done by machine, but that’s not the case. Since we’re also changing our information architecture — adding a lot of semantic structure where previously there was only presentational information — it takes an informed human being to know how to convert an older page into its newer format. A real person is needed to understand what a piece of content is really about, and what it isn’t about but merely mentions in passing. We also need people to look at the new keyword pages (for example, the one on immigration and asylum, or the page on Kosovo), which are populated mostly automatically, and for them to highlight the most important content: the most important content won’t necessarily be the newest.

This work had been going on for many weeks before the weekend launch. The January software release brought in some tools refinements to help them tidy up final loose ends (and no doubt some more tidying will happen over the next couple of weeks).

You can see from this that it’s about much more than mere software. The software enables people to do the editorial work, and the editorial work has been going on for some considerable time. Everything they’ve done has also been tested and previewed, which allows them to see what our external users would see were it to go live. Again, this exercises the software fully, but only within the internal network, before it’s exposed to the outside world.

3. Rehearsals

The work for the weekend launch is mainly running a lot of database scripts to make various new URLs live and decommission old ones. The reason this is such a huge launch is that there’s over ten years’ worth of news content to expose, as well as new functionality and designs.

We couldn’t trust the scripts to work first time (of course), so we spent a lot of time copying the production content into a safe environment, and rehearsing the process there, with real data. We needed to be sure not just it would work, but also how long it would take (considering any hardware differences), and change the scripts and process accordingly.

Guardian.co.uk favicon4. Launch

Finally, after all the rehearsals, the real deal. The work to run the database scripts and raise the curtain on various features ran over Saturday and Sunday, but it was calm enough and organised enough that the team needed to work only slightly extended working days.

So the big launch was a culmination of a huge amount of effort, and the weekend work was after an awful lot of practice. There were a couple of sticky moments, but nothing the team couldn’t recover from in a few minutes. As one of the developers remarked towards the end of Sunday: “The fact that today’s been really tedious is a good thing.”

What we can see now includes

What’s next…

We’ll be ironing out a few things over the next few days, but everything’s gone to plan for now. And then, as Emily says, there’s still sport, arts, life and style, and education to do.

The unstoppable urgency of web development

While I’m usually proud of the work I’m involved with, I’m rarely happy for long. There are always ways to improve, and I’m usually dissatisfied by one unmet ideal or another. Almost since I started in this field I’ve been vexed by how much of web development is “urgent” rather than “important”. This is not merely of theoretical interest; it vexes me because dealing with important issues gets you to something of lasting and strategic value, while dealing with urgent issues gets you to the next day. It’s that unmet ideal of achieving more long-lasting value that causes me to think about this.

I want to give some context around the claim that there’s more urgency in web development, and then I want to offer some ideas about why this might be. If you want the brief version, I think what makes the web generate such urgency is that…

  1. Updates are cheaper;
  2. Problems hurt less;
  3. Competition is more visible;
  4. Boundaries are blurred; and
  5. It’s easier to have more stakeholders.

But first, some context.

Picture will go here when I have timeSome context

Although I do think web development has a surfeit of urgency, it’s fair to say that “web development” is not a clearly defined area. So much software today has some link or other to the web, it’s usually unfair to say definitively that something is or is not web development. So it’s a matter of degrees, and looking back on the software development I’ve been involved with I’d say the more web-oriented something has been the more I’ve had to deal with urgent issues, and the less web-oriented something has been the more I’ve had to deal with important issues. The most urgency comes when working on websites, and the least when working on applications which may or may not use the web for a bit of minor communication.

This issue of urgency also seems to be the perception and expectation of non-technical people. This is how we get the phrase “Internet time” and why for years I’ve heard people saying “but this is the Internet” as a justification for tight deadlines. This is something that continues today, even after the unsustainable boom and bust of the dot.com years.

When urgency is too pressing we risk doing a bad job, and releasing something which isn’t up to scratch. Of course, when urgency isn’t pressing enough we lose ground to our competitors. And of those two scenarios it’s the first one, doing a bad job, which is easiest to spot: a bug bites someone on the nose and you get to hear about it pretty fast. The second scenario, losing to the competiton, happens slowly over months or years, and by the time you slip into obscurity so many events have occurred it’s hard to pinpoint exactly where you went wrong.

Urgency is no bad thing, then, but I continue to be bothered by its dominance in web development. Too much urgency leads to a loss of quality and a reduction of strategic actions. Why does this happen in web development more than elsewhere? Here are some thoughts…

1. Updates are cheaper on the web

Rolling out a new version of a website onto a server shouldn’t be regarded as a trivial affair, but it’s a lot easier than other means of releasing software. So it’s easier to release a slightly rushed product if we know the only copies anywhere in the world live on servers which are entirely under our control.

Slightly more costly than updating our own servers is the job of the IT department which has to update an application on everyone’s desktop. Here, not only do you have to be sure the desktop machines are all switched on and ready to receive the software, but an error for one person could mean they cannot work at all the next day. More costly than that is the job of burning, boxing, and shipping a DVD to hundreds of retail stores.

For both those cases, and for many other ways of releasing software, the cost of repetition and the cost shipping something substandard is very high. But in many web applications, where the cost of distribution and the cost of error is relatively low, the consequences of problems that come from an urgent release are much less costly.

2. Problems hurt less on the web

As users, our relationship with websites and web applications tends to be less engaged than with most other applications. So although we all know undue urgency leads to more faults being released, it’s perhaps more forgivable on the web.

If I’m using a website and I encounter an error, I know the problem — the website — is theirs. I can come back later or try an alternative (there are always plenty of alternatives on the web). Either way, I’m probably not going to get very upset.

By contrast, if I encounter a problem with software running on my machine then any error is going to hurt me much more: that’s a problem with my software… it’s causing my machine to go wrong… I went to the trouble of installing that software and now it’s giving me problems… this is not good. Even if the software was installed by my IT department it’s still my software, and it’s core part of my daily work — finding a workaround is going to be difficult.

3. Competition is more visible on the web

One of the drivers of urgency is beating the competition, and if you and your competitors all work on the web, then they are likely to make (cheap) incremental releases, and you will notice almost every one.

If competitor A releases a new feature in week 1 then you’ll notice and ask how you can do that yourselves. If competitor B releases another new feature in week 2 then you’ll notice it and ask how you can beat that, too. By the time competitor C releases a third thing in week 3 you’ve got the pressure of the three things being delivered by your competition and nothing to show yourself.

Things are different as you move offline. Consider software companies in a competitve marketplace that see software ship features in batches. When your competition ships their new release with new features you’ll be able to assess all those features as a whole. Of course there will still be intense pressure to deliver, but your view will be more rounded and decisions of importance will play a greater role than decisions of urgency.

Consider also the internal team which delivers software which is internally-facing. Your life in this team will be no less demanding than in any other, but one thing you’ll be spared is much pressure from competitive teams. If you’re delivering the next version of the payroll system then there will be pressure of deadlines, there will be pressure of budget, but you’re very unlikely to get an unexpected new requirement due to the payroll software team at your competitor suddenly releasing their new e-mail alert module.

None of this is to say that competition is more fierce on the web. That may or may not be the case, but it’s not what I’m interested in here. The point is that the competition seems more fierce and is much more visible. This encourages urgency.

4. Boundaries are blurred on the web

A wordprocessor helps you write documents, a media player plays audio and video, but a website… a website doesn’t have clear boundaries.

Although I wrote above of competition, exactly who is a competitor on the web is not always certain. I once worked with an e-commerce music retailer who suddenly decided they needed to provide a web mail service for their customers. I’ve worked with a law firm who wanted to turn their brochureware site into a recruitment platform. Yahoo! evolved from a directory into a portal, embracing almost everything they could while the world struggled to understand what a portal might not be. Meanwhile, stepping away from the web, I don’t know of a spreadsheet which carries a discussion forum or an HR system which recommends movies.

While all this clearly accounts for the quantity of demands that occur in web development, it also accounts for the urgency because that steady drip-drip-drip of features that comes from our competition also comes from those who aren’t direct competition but are just out there on the web. The drip-drip-drip becomes a trickle (and maybe even a flood) pretty quickly, and the pressure to deliver new things is even greater.

5. It’s easier to have more stakeholders on the web

Because the web means so many things to so many people — contributing to those blurred boundaries — many more people within an organisation will consider themselves stakeholders. Marketing people see it as a marketing tool, the recruitment team see it as a recruitment tool, the sales team see it as a sales tool, and so on. And of course each one of them is right: it can be all these things.

With so many diverse stakeholders, we can expect so many more pressing deadlines, and hence much more urgency. More stakeholders also means communication and prioritisation is more difficult. It’s difficult to get everyone into the same room, it’s difficult to bring everyone to the same point of understanding, it’s difficult to weigh completely different kinds of requirements against each other. It doesn’t matter that everyone is working for the same company and therefore, at a higher level, the same goal. That goal is usually at too high a level to make a difference to everyone’s day-to-day needs.

The least pleasant example of this I remember is a colleague at a previous company who was the project manager for one of our clients. Although he was their single point of contact they managed to have several single points of contact, none of whom were very good at speaking to the others. He found he had responsibilities to the marketing manager, the distribution manager and the IT director. They all had pressing demands, and he had no contact with anyone who could bring them all into line. Indeed, that seemed to be by design, as we suspected the client used this as a way of squeezing us as a supplier. The relationship did not end happily. Although that’s a very extreme example it does highlight the kind of difficulties of having many stakeholders.

A final word

Cheaper updates, less damaging problems, more visible competition, blurred boundaries and more stakeholders. This tends to be the world of web development, and therefore a world in which urgency plays a more dominant role.

Yet we can think of web development projects in which importance dominates urgency. One which springs to mind is an online banking site: this is web development if ever anything is, but one in which doing things quickly will always be trumped by doing things properly (whatever “properly” may mean for that bank). But this is also an environment where many of the criteria I’ve listed don’t hold. Most obviously updates will be fairly costly, any problems will be very damaging, and the boundary of what the site does is very well defined.

So the criteria I’ve set out above are more to explain than define. Web development tends to take place in an environment that’s different from many other software development environments, and I think it’s those features of that environment that I’ve listed that explain why urgency predominates.

I said at the beginning this vexes me because urgency causes visible problems, and eliminating urgency would seem to help eliminate those problems. But I also said I’m usually dissatisfied with one unmet ideal or another. I’ve also worked in environments where there is a distinct lack of urgency. That, of course, leads to waste and stagnation. And that’s really, really, vexing…

“But why didn’t you write it in Ruby…?”

I thought I’d answer a question that comes up from time to time about the Java development and redesign work going on for the Guardian Unlimited site: “Why didn’t you write it in Ruby?”

I was reminded of this question because of an excellent post over at Manageability.org in which ceperez wonders aloud whether dynamic languages can scale. The post contains excellent thumbnail sketches of various languages, and the suggestion is that different languages cultivate different habits which have different consequences as an application (or project) evolves. The flipside of this question is Nati Shalom pondering why Java isn’t used more for large scale web applications. Nati’s post is one I’ve kept around for some time, wanting to follow it up, and so this post finally does that.

The question of why the Guardian Unlimited redesign isn’t taking place in Ruby is sometimes asked of me, but most often asked of any of our architects, often following a presentation of our work. Here are some answers.

Developers discuss Java and Ruby1. A personal answer

My own immediate answer to the question is usually a reaction to the fact that it is too often asked by certain kinds of people. These people are the Kool Kidz who have picked up Ruby on Rails over a weekend, created a blog in 15 minutes, sneaked a Ruby script onto their company’s servers, and are so delighted with their new toy that they think it should be the basis of the multi-million pound invoicing system currently wanted by the Accounts Department, which they’d quite like to build if they can use Ruby but which they have no intention of sticking around and supporting for the next ten years.

Unfortunately my personal answer to these people contains too many expletives to be included on a family blog.

Just as well we’ve got some other answers, then…

2. Confidence

Ruby might be a terrific language, with good support, and it might be really reliable. But the redesign work that’s going on now produced its first public release in November 2006. So the work will have started a few months before then, and the planning a while before that. Let’s round it out to January 2006, two years ago from now.

Casting your mind back two years you may recall that Ruby was an up-and-coming language, but it hadn’t really been proven running large scale websites. Sure, 37signals were working with it to great effect, but that’s only one case study, and not of a typical company. If you’re going to sink someone else’s money (that is, your employer’s money) into a big project you have a responsibility to ensure you’ll get results. You’ll want to be sure there is sufficient expertise in the marketplace to call in help when you need it, you’ll need to see several other similar projects that have succeeded with it, and you’ll probably want to know of a few projects that have failed with it so you know what not to do. With the state of Ruby at the time there just wasn’t that weight of evidence that supported its use for our purpose.

3. Scalability and socialness

Finally, a very important factor in our consideration was a kind of scalability, and what one of our architects calls socialness. It’s the scaling of the team.

There’s a world of difference between four-or-so developers writing some software and several dozen developers writing it. The day-to-day mechanics of writing software changes significantly when you’ve got so many developers that…

  • there are too many to fit round the same table;
  • you can no longer rely on saying something aloud and being heard by everyone;
  • you couldn’t possibly have recruited them all if you had wanted Indentikit developers who would all naturally solve all problems in exactly the same way;
  • any code you write is likely to be picked up next by someone you’ve not happened to speak to for more than a fortnight;
  • any code you pick up is likely to have been written by someone you’ve not happened to speak to for more than a fortnight;
  • it’s only to be expected that the people you end the project with won’t be all the same people you started with.

When all this is the case, then code consistency becomes hugely significant, and just relying on people’s own technical and communication skills isn’t going to be sufficient — you need your software to keep you on the straight and narrow. We weren’t comfortable that a language with the fluidity and flexibility of Ruby would provide that.

Postscript

I’ve just made a quick calculation: If quality is compound, and you’ve got 25 developers who each manage to code to 95% of your desired standards, then the quality of your codebase becomes 0.95^25, or 28%. Okay, it’s a pretty spurious calculation, but it goes to illustrate the point that quality is more fragile across a larger team.

In the end, the “why not Ruby” question comes down significantly to historical reasons and human reasons.

And what would we choose if we had to make the decision today? Well, we don’t have to.

Lightweight versus heavyweight: The cost is in the management

A recent conversation with a colleague got me thinking about so-called “lightweight” systems, and when they become more trouble then they’re worth. He was frustrated by some problems he was having; even more so, he explained, because he thought he was dealing with something that was “lightweight”. It’s a seductive word, and sometimes — as with other forms of seduction — when you get more involved than you should things can get a bit sticky.

This article is an attempt to explain what lightweight really means, both in terms of benefits and drawbacks. There are also a couple of comparative examples from my own experience.

A lightweight system (plus management support)Lightweight doesn’t mean simple

People often mistake “lightweight” to mean simple or quick. But this can’t be right, because everyone wants simple and quick, and if it really meant this no-one would use anything else. Every website would be rewritten with the lightweight Ruby on Rails and every application would be sitting on top of the lightweight SQLite database. Who wouldn’t? Who doesn’t want simple? Who doesn’t want quick?

Lightweight is often good, but it must have its tradeoffs, otherwise other technologies wouldn’t exist.

From the examples below I see lightweight as offering low cost in return for low demands, high cost for high demands. Heavyweight is disproportionately high cost for low demands, but low cost for high demands.

Lightweight carries low inherent management costs. But some situations require a high degree of management control whether you like it or not. That means that if a lightweight system needs to scale up you have to wrest management from it and maintain it externally. If you can do that then the lightweight system continues to work, but if the lightweight system will not relinquish management control, or if you don’t have the discipline to keep the management going, then it won’t be effective in the long run. By contrast heavyweight systems impose management and structure of their own. This is good if you’re going to need it, as it takes the pressure of discipline off you, but it’s not effective if you didn’t need that management structure in the first place.

To illustrate this, here are a couple of lightweight/heavyweight comparison case studies…

Language example: TCL and Java

TCL is a lightweight language. You get to write Hello World in one line, it doesn’t force much structure on you, and it’s pretty relaxed about how it’s written.

TCL is so good, in fact, that was the basis of the original Guardian Unlimited website. We built Ajax-style tools with it before Ajax was known as a concept, we generated our front page from it, we used it to integrate with our ad server.

But as our site grew the language didn’t scale with it. Clever shortcuts implemented by earlier developers confused newer developers because they obscured the purpose of the code. The lack of an imposed structure meant every foray into older code involved learning its idiosyncracies from scratch. Development slowed down as we worked around older code. And when we wanted to redesign the website we found that through years of lightweight flexibility we had allowed ourselves to be tied into knots: it would be more effective to start again than to work with what we had.

In fact, for the most part we’re now using Java…

In contrast to TCL, Java is pretty heavyweight. Not only does Hello World require three lines (excluding any lines with just braces), but its philosophy of structure and layering percolates through from the core language to most of its add-ons. For example, to parse an XML document you have to drill through two abstraction layers before you can find the parser.

One Java framework that maintains this ethos is Hibernate, used for database access. Its architecture is complicated, and as usual this is to offer flexibility without relinquishing manageability. Recently a forthcoming release of the Guardian Unlimited website was failing its pre-production performance tests. Our developers tracked down a major cause of the problem to an inefficient query within Hibernate. They extracted some of the query’s logic up into the application layer and simplified what remained, rebalancing the work between the application and the database. Problem solved, performance restored. What’s relevant to our story is that the developers did this entirely within the archicture of Hibernate, so they didn’t compromise the design of the application and therefore didn’t add complexity.

CMS example: WordPress and the GU CMS

Over on ZDNet Larry Dignan extolls the virtues of WordPress and says, effectively, “What have big content management systems ever done for us?”

WordPress is the lightweight CMS I’ve chosen for this blog, and I’m very happy with it. It’s easy to install, requires almost zero maintenance, and lets me focus on the writing. And yet I’m a strong advocate of the home-grown CMS we have for our journalists and subs on the Guardian Unlimited site. Is lightweight not good enough for our journalists? What has a big CMS ever done for us?

Well, I just looked at a current article on guardian.co.uk: “Ministers ordered to assess climate cost of all decisions”. It was created with our big CMS. What’s there that WordPress couldn’t deliver?

For a start it’s got a list of linked subjects down the side, which aren’t the same as WordPress’s tags because they’re tightly managed to ensure consistency and reliability. These subjects are also categorised, so Pollution and Climate Change are subjects under Environment, while Green politics is a subject under Politics. As I write this, I note also that the pages for Pollution and Climate Change are designed differently, with Climate Change being more pictorial and feature-led. Subject categorisations and subject-specific designs are beyond what WordPress’s tags do.

Okay, so apart from the linked subjects, the categorisations, and the subject-specific designs, what has a big CMS ever done for us? I suppose it’s worth mentioning the related advertising, which as write includes a large ad for environmentally-friendly washing liquid. There are other contextual commercial elements, too, such as the sponsored features, links to green products and books, and offers of reducing energy bills and offsetting carbon emissions. And there are related articles and related galleries. And details of the article history, listing when and where it was first published, on what page and in what newspaper section.

Okay, so apart from the linked subjects, the categorisations, the subject-specific designs, the related advertising, the contextual sponsored features, the links to relevant products and books, the complementary offers, the related articles, the related galleries, and the article history, what has a big CMS ever done for us?

Well, I suppose it is serving to over 17 million unique users a month…

I’ll stop now. The point is a lightweight CMS such as WordPress could probably do any one of these things, with a bit of work. But it isn’t designed to do anywhere near all of them. And each time it’s changed to do one more of these things the more it is moved away from its core architecture and it gets closer to a point of paralysis, where nothing functions well anymore because no part of it is doing what it was designed to do. A bit like the TCL example.

Looking back

Reviewing these two examples, it’s clear that the lightweight systems became, or would become, very costly when they were pushed beyond their initial expectations. In both cases the corresponding heavyweight systems came with their own (heavy) management structure, but that management structure ensures lower running costs.

In the Hibernate example our software maintained its architecture after we’d made our performance change; anyone looking at this new code would be able to rely on previous knowledge to understand what was going on. By contrast, anyone coming fresh to a snippet of old TCL code would be starting from scratch, regardless of how much of the other TCL code they’d seen.

Similarly, the large-scale content management system at GU is internally consistent, despite its vast range of features and functionality. Once someone has learnt the principles (which, admittedly, are non-trivial) they can get to work on pretty much any part of it. Pushing WordPress to do that would have created a monster.

Lightweight systems take the management away from you. And that’s ideal, as long you don’t need that manageability.

We (heart) UAT

When do we need user acceptance testing? And when can we get away without it?

User acceptance testing (UAT) is when your software goes in front of the user to get final sign-off — and when they ask for changes if not. In theory you shouldn’t need UAT at all (didn’t they tell you what they wanted? Weren’t you listening?), and indeed perhaps you can sometimes get away without it.

Testing a search engine rigorouslyThis argument is stronger if you’re considering UAT throughout the project, not just at the end. This would be the case if you’re UATing individual features, or individual deliveries in a culture of more and smaller iterations. The arguments are: more deliverables mean more UAT means more expense for the customer; smaller iterations mean a reduced chance of getting it wrong; frequent deliverables give the customer more opportunities to change anything they don’t like anyway.

However, even with small, frequent deliveries there are circumstances when it’s more important to do UAT. Looking back on some projects I’ve worked on, here are times when it was, or would have been, a good idea…

1. When the output of the software is subjective

I once worked on a project that asked what bands you liked and then based on that recommended albums by unsigned bands. This was a big leap away from the “people who liked that also liked this…” kind of recommendations, because the unsigned bands won’t have had a significant userbase, so we couldn’t rely on a self-generating wisdom-of-crowds mechanism. Plus, of course, music recommendation is highly subjective, and it was our clients who were the music experts — we implementers were mere software developers.

In this case you can see it was key for our clients to play with the system and be sure they were comfortable with it. If they didn’t then it would have meant tweaking the algorithm, not a complete software rewrite of the software. In the event something rather unexpected happened. While our client’s representative was very happy with the output he also found it rather disconcerting. He felt somewhat disenfranchised from the system because it seemed so mysterious. He realised that he couldn’t demonstrate it in front of his colleagues and their investors without having an answer for the suddenly inevitable question: how does it work? And once this was explained to hiim he became much more comfortable again.

If the output of the system had been much more objective and predictable this instance of UAT wouldn’t have been so important. In the end it threw up an important new acceptance criteria for client which we were quickly able to address.

2. When the software has a costly userbase

One of my past projects involved creating a user interface for a property data entry team. Individuals on the team were, as you’d expect, low-paid and largely interchangeable — after an hour of training you would know everything there was to know about the system. Your sole function was then to spend hour after unsociable hour entering information about properties you could never afford to live in for people you would never want to live next to.

But although the daily cost of individuals on the team was considered low, the cost of the team as a whole — and the value they were bringing to the business — was very high.

Problems with the user interface in this case would have had a knock-on effect across the whole team of 20 or 30 individuals, and cost their employer dear. It was key for someone to check not only that the system was responsive, but also that there were intuitive keystrokes across the data fields, data entry was reasonably forgiving, tabbing between fields happened in the order that data was presented in, and so on. In this case, the time and early feedback from one user at the vanguard saved time and cost 20 or 30 times over.

3. When you don’t have a QA department

A pretty obvious one, but if you don’t have people dedicated to quality assurance then those internal people who take on that function (probably the lucky developers or irritated account manager) will be too close to the software and too distant from client to pick up on all the issues that are important to them.

A good percentage of the projects I’ve worked on have involved integrating search engines. And in almost every instance when testing has been left to the development team they’ve alighted on one or two key phrases to use as test cases and considered that sufficient. By contrast, the end user to whom this is important will have half a dozen phrases that are relevant each day — and they’ll involve accented characters, non-ASCII characters, mismatched quote marks, and so on. And that’s before we get to the results page. Without effort the expert user has given the search engine a good workout — often raising difficult issues for the developers.

Actually, even if you do have a QA department then there’s a case for UAT, because more often than not the QA team will test against written criteria. There are very often criteria which, with the best effort in the world, just don’t get realised or written down at the requirements stage. And in these cases it’s only the end user who can really tell if it’s right.

4. When you lack detailed requirements

Another fairly obvious situation, but if you’ve not had a good bit of customer input at the start of the project, then UAT is going to be even more important later on.

A sales person I worked with once had a cunning plan. We’d already produced an e-commerce site for one of our clients, and he decided to sell a duplicate system to one of their rivals. It was good plan on paper. There was no contractual restriction for us; the new client would get their site at a relatively low cost; they’d get it quicker than usual; and we’d only have to reskin it. The deal was done over a nice lunch.

You can guess what happened in reality. When they saw what we imagined was a near-complete version they wanted some changes. The checkout system wasn’t quite to their liking; their product catalogue didn’t have quite the fields we were expecting; the order placement system didn’t align with what they had internally; and so on. It cost everyone more than they had anticipated and it was delivered later than anyone would have liked.

Certainly this is a strong case for robust specifications — or at least understanding what you’re getting into before you get into it. But it also demonstrates that lack of early information cost us dearly when the information did come along later. We hadn’t made time for UAT, which is why we delivered the final product later than planned. Fortunately for everyone there was no externally-driven deadline. If there had been then we’d have been obliged to deliver something which the client wasn’t happy with.

5. When the inputs to the software are difficult to specify

MediaGuardian.co.ukFinally an example that brings us right to the present.

The current work to refresh the Guardian Unlimited website is much more than applying a lick of paint. We’re also producing new tools for our reporters and new ways they can tell their stories. But telling a news story is not simply a matter of filling in a few boxes and clicking Save. To make a story relevant requires a lot of hard training. To edit an entire news section additionally needs an awful lot of experience and good intuition…

As I write, MediaGuardian.co.uk is leading with three big stories and a podcast, plus some smaller items headed “More media news”. The stories have between three and six sublinks which indicates the depth of related news that the editor has available. Yet at the same time the Technology section is showcasing six stories, all with a bit of blurb, but only one of which has sublinks (to a video and a gallery). In both cases the editors are using the same system, but the page layout has to balance itself out regardless of the highly variable input.

But though the input is highly variable it is not without bounds. A section would never be showing just one story, and an editor would almost certainly not try to highlight twenty stories and claim they had largely equal weight. The only way to really know the bounds in which the system is supposed to reasonably work is to sit an editor down with the tools and let them lay out pages in response to a real day’s stories. No amount of requirements analysis and QA experience can substitute for a real journalist responding to a real day’s events.

And that’s the beginning…

No mechanism should be applied blindly; there are times when user acceptance testing is more appropriate, and times when it is less appropriate. The important thing is to be aware first that UAT exists, and second that there are criteria which you can apply to assess how valuable it can be.

Amazingly, some people aren’t motivated by efficiency

Staggering though it may be, it turns out that people are different. It also turns out that certain kinds of people are different to other kinds of people. And a corollary of this is that people who aren’t software engineers tend to have a different perspective to those who are.

For example, I spend a lot of my time trying to maximise our efficiency. Maybe this is because I’m a techie kind of person; I see the same motivation among the project managers, developers, and others in our software team. But very often we face the prospect of having to start a piece of work without having all the details available to finish the job. Maybe we’re waiting for information from a third party, maybe the visual design isn’t complete, maybe a decision still needs to be made by people elsewhere in the organisation. Whatever it is, we’re faced with the prospect of starting a piece of work knowing that we may not be able to complete it without an interruption.

We built the car very efficiently...This is clearly going to be frustrating to my techie, left-brained approach to life. It undermines my efforts to be efficient, organised, anally retentive, and generally less fun at parties.

I know it would frustrate others, too. Agile advocate Simon Baker recently railled against organisations that didn’t make product owners sufficiently available to provide the relevant feedback and information. He wrote

If the project is vital to the business, then the company can always find a way to provide a full-time and colocated Product Owner. If they say they can’t, it really means they won’t. Quite simply, they’re not prepared to do what is necessary to achieve it, and frankly, if they’re not going to take the project seriously why should you?

Ouch. I’ve felt the pain that Simon felt when he wrote those words, but we shouldn’t rush to make harsh judgements on business experts who are facing pressures of their own.

We software people can talk all we want about our methodologies, but sometimes we need to wake up to the cold, wet slap of reality. The fact is software development is not the be-all and end-all of most businesses — far less is efficient software development methodologies. Sometimes it’s more important to be working on the most important thing inefficiently than it is to be working on the second-most important thing efficiently.

At Guardian Unlimited we motivate ourselves by measuring velocity — the number of units of work we complete in a given period. But if we’re not careful we can focus on that too much and be in danger of missing the bigger picture.

Some time ago I was speaking to one of our internal customers, explaining why developers pushed back on incomplete specifications, and the motivation of being efficient and achieving target velocity. “I don’t give a stuff about velocity,” she said, “I just want the thing built.” The point is well taken. Sometimes we need to remember what the word “customer” means.