I’ve just finished a fortnight’s holiday which I (foolishly) spent mostly in front of a PC developing a never-ending little application. But unexpectedly, despite the trivial nature of my project, I rediscovered a number of important lessons more usually associated with serious application development.

The software I’m writing is a just a little Firefox plugin. I’ve been fiddling with it for so long we’re almost onto the second major release of Firefox since I started, and yet it’s probably not much more than a couple of hundred lines long. You can see it really is a minor enterprise. Despite this it’s been quite a surprise — quite a shock, even — to be reminded of some industrial-strength truths in a small and personal environment.

And they are…

1. Damn, writing software is difficult

What we non-developers know about software we know either by observing or by talking to those who do it. But anyone who wants to be trusted won’t complain about their job or bore you with details they know you don’t want to hear, and so you’ll never hear about everything there is to know about software development, even when you ask.

So one thing I discovered — again — is that writing software is really difficult. Sometimes I was flying, but more often I was crawling: piecing together information from different sources, trying to understand what was possible, learning all kinds of technologies (Javascript, XUL, XPCOM), trying use them well, but more often trying to get them to work at all.

It’s made me respect all the more the people I work with every day who make it look so easy. Every line you write is pure logic and needs to work 100%. This is not like writing a letter or a sales proposal, where a sentence that’s only 95% perfect is more than adequate. This is like writing a legal contract from scratch for a particularly unpredictable world. Every line needs to be watertight.

2. Simple design really is difficult

Simple design is another way of saying that the software should be easy to work with and modify. You’d think this would be hard to get wrong, particularly with something which is being built as new: surely you just add small, simple pieces one at a time, with each one adding the next feature on the list. What could go wrong?

I found out first hand that simple design is difficult to achieve, even in very simple scenarios. For example, my application has a class which handles the user’s preferences. It needs to be initialised for when the application starts, and it needs to take and save the preferences when requested. All this takes place within a single object. So I wrote an initialisation function which set up the initial preferences according to how it’s been configured, I tested it, and all was well. Then I added some functions to take updated values and save them, tested those, and all continued to be well. Total: about 20 lines of code over four functions.

Then I ran the whole in an integrated environment and weird errors started occurring. It took me about two hours to figure out what was going on: it was a strange combination of unexpected start-up values, Firefox calling the initialisation function at unexpected times, and some confusing logic of my own which was supposed to protect double-initialisation. In the end I decided the best solution was to throw out the whole idea of initialisation. Now the object just takes and save preferences, and you can do that when the application starts if you want. Total: 5 lines.

The point of this is that an apparently simple and obvious design was actually too complicated to sustain. I’m very pleased that the solution was to delete lines and simplify. But I could only do this so easily because I had complete control of the code. In other circumstances there might have been other systems which were relying on that initialisation code (however flawed it may have been), and I might have had to add to the existing complications to solve my problem; or I might not have had the time to take a fresh look at the code and simply built around the flaws out of a sense of fear of touching the wrong thing.

This is a tiny example from a tiny piece of work, but it showed me how easy it is to go wrong with a design, and how easy it is to produce software that is complicated, hard to understand, and time-consuming to fix and evolve.

3. Learning a language is more about culture than syntax

I sometimes get CVs from people who claim to know about 10 programming languages, and I’m always doubtful. Just because you’ve written an application in a language it does not follow you’ve done it well. Knowing the syntax is only the first step. You also need to have good knowledge of any libraries, and finally you need to know how to work with the grain of a language. This means you’ll use different idioms, and structure your solutions in different ways.

In my own case I’ve been writing Javascript, but it stinks because I’ve tried to use it like Java. I’ve been stuck in my old Java ways, like creating classes and carving out a deeply-nested namespace. It’s Javascript, Jim, but not as we know it. It works, it makes sense, but it looks clunky and… well, it just feels wrong, dammit.

Javascript is a prototyping language. I can even tell you what that means, but only with my head, not with my heart. I use the prototyping as a hoop to jump through to get it to do the Java-y things that I know I shouldn’t be doing in the first place. Being a prototyping language doesn’t mean Javascript is a second-class language, or a dumbed-down Java. It means it’s a different kind of language to Java and should be treated as a first class language with its own ways of getting things done.

It’s a cultural thing, and you can’t claim to really know the language if you don’t operate comfortably in its culture. I don’t really know Javascript.

4. How did I ever live without automated tests?

Possibly by not spending my holidays sat in front of a PC. But aside from that, I continue to wonder at the marvel that is automated testing, and unit testing in particular. To be able to implement a change and not have to trouble your brain about the consequences is very liberating, allowing you to move ahead with confidence. It does take some work to set up the environment, but the results are worth it.

That said…

5. Your automated tests won’t cover everything

In one of my functions I unexpectedly found a truck-sized hole which had gone undiscovered despite seemingly comprehensive automated test coverage. (A loop which had a “break” when it should have had a “continue”, meaning great swathes of actions got skipped in most circumstances.) I only discovered this through integration testing (which is the fancy name I’m using for what was really “trying it out”), and found that a quirk in my unit testing setup had caused the mistake to be missed. Once I had found the cause I adjusted the main code to be more predictable and put in an automated test to trap the error, but it was only discovered through real hands-on testing.

6. A strong IDE sustains motivation as much as anything

Although I was using Eclipse for development, when working with Javascript it really doesn’t offer the comprehensive support you get with Java. Because Javascript is a dynamic language, and no doubt also because of the state of JSEclipse, there’s very weak support for code completion, refactoring and so on.

The consequence of this was a loss of code-writing speed, but much more than that I was suddenly able to see how easily a weak IDE allows bad software to be produced. There were many occasions when I knew that I should tidy up or refactor something, but was then suddenly hit with a premonition of the tedious steps I’d have to go through: working out which files to pick on, the manual search-and-replace, checking the context before I made a change. I had to force myself to get on with the tidying up despite knowing how painful it would be, focusing on the long-term results, and safe in the knowledge that for this little personal project I didn’t have a deadline.

It became clear to me that so many people must come under a constant barrage of pressure, with only their current strength of character to defend against the pressures of deadlines and short-term wins. It’s inevitable that too often they will give in to those pressures, leaving cumbersome code building up, and ultimately gumming up the works of the system. A strong IDE removes the barriers to those virtuous tasks of improving design and allows you not only to do your job, but to do it well.

7. Estimation is difficult

Which is just an excuse for the shameful truth: all my estimations were out by a factor of four. This is embarrassing because it’s not as if I’ve never written software before.

In retrospect the mistake I was making was to look at the component parts of a task, guess how long they’d each take, and add them up. What I should have done was try to take the forthcoming work, identify similar previous tasks, and from past experience see how long they actually took.

At the end of the day experience is good, but it’s how you use it that really counts.

8. You can’t know all your requirements up front

This is familiar to anyone who’s bought into Agile, but it hits home hard when it’s you who’s the user setting the requirements.

I’ve written countless requirements specifications in the past (in 60-page documents, on task cards, wherever) — I thought I really ought to be clear-thinking enough to know my own requirements up front. Wrong again. As the UI came together, as one idea sparked off another, and as I had chances to step away from the code to think about things from a distance, I started to see that my feature set was really rather disjoint — almost random. These were moments of clarity that on the one hand caused me to add requirements, but in doing so I was recasting the software with a new perspective. I was starting to see what the software should be doing, which was not quite the same as what I’d started out on.

Release date

Fortunately I’ve not sent out a press release announcing a release date, held a press conference, or hired the London Eye for a glitzy media event. I’m just writing software for fun, and at this rate it’s probably not going to ever see the light of day. But even then, it’s been startling to find that the germs of some of the Big Ideas of software development are still present in the smallest of projects.

There were very few moments for me during QCon London 2008 of earth-shaking enlightenment — if any. But every hour of the three days of the conference there were insights and guidance that could be tucked away, and reused later to save hours, days or weeks of time elsewhere. Snake-oil salesmen where thin on the ground, and instead there were dozens of people saying one or both of:

• This is what we did; and
• This is what you can do.

No magic, no silver bullets, but plenty of solid advice and experience.

A good example of both of these was Randy Shoup of eBay. He had nothing to sell (other than the good name of eBay, perhaps) and his presentation was very clearly constructed to show their principles of scalability, and some concrete examples of how these work in practice. You probably wouldn’t use their periodic batch processing method to generate recommendations — if only because it’s odds on you don’t have a recommendation system — but you could take the overarching principle of “async everywhere” and apply that to the next scalable application that you need to work on.

Even the very specific presentations contained valuable points that could be generalised and reused. For example, Matt Youill and Asher Glynn of Betfair talked through how they scaled the transaction processing on their servers by a hundred-fold. Guardian.co.uk doesn’t need that kind of throughput, so the details were primarily of intellectual benefit. But a key practical lesson was how they approached the problem: by presenting it to industry players as a challenge carrying great kudos to the winning company.

All of this was summed up very nicely by the team from BBC News: John O’Donovan, Kevin Hinde and Ross Heritage. They were asked how they managed performance testing for the iPlayer. John spent a few moments describing some of the techniques they employed, but got to the point when he realised the audience really wanted some eye-opening enlightenment which he didn’t have. At this moment Kevin stepped forward and said straight out “There’s no secret sauce”. Indeed not: they just work hard and stick to strong principles.

QCon offered little in the way of secret sauces, but it did contain dozens and dozens of great ingredients you could take away and use to concoct your own wonderful dishes.

And with that analogy pushed to breaking point, I think we should leave it there.

While I’m usually proud of the work I’m involved with, I’m rarely happy for long. There are always ways to improve, and I’m usually dissatisfied by one unmet ideal or another. Almost since I started in this field I’ve been vexed by how much of web development is “urgent” rather than “important”. This is not merely of theoretical interest; it vexes me because dealing with important issues gets you to something of lasting and strategic value, while dealing with urgent issues gets you to the next day. It’s that unmet ideal of achieving more long-lasting value that causes me to think about this.

I want to give some context around the claim that there’s more urgency in web development, and then I want to offer some ideas about why this might be. If you want the brief version, I think what makes the web generate such urgency is that…

1. Updates are cheaper;
2. Problems hurt less;
3. Competition is more visible;
4. Boundaries are blurred; and
5. It’s easier to have more stakeholders.

But first, some context.

Some context

Although I do think web development has a surfeit of urgency, it’s fair to say that “web development” is not a clearly defined area. So much software today has some link or other to the web, it’s usually unfair to say definitively that something is or is not web development. So it’s a matter of degrees, and looking back on the software development I’ve been involved with I’d say the more web-oriented something has been the more I’ve had to deal with urgent issues, and the less web-oriented something has been the more I’ve had to deal with important issues. The most urgency comes when working on websites, and the least when working on applications which may or may not use the web for a bit of minor communication.

This issue of urgency also seems to be the perception and expectation of non-technical people. This is how we get the phrase “Internet time” and why for years I’ve heard people saying “but this is the Internet” as a justification for tight deadlines. This is something that continues today, even after the unsustainable boom and bust of the dot.com years.

When urgency is too pressing we risk doing a bad job, and releasing something which isn’t up to scratch. Of course, when urgency isn’t pressing enough we lose ground to our competitors. And of those two scenarios it’s the first one, doing a bad job, which is easiest to spot: a bug bites someone on the nose and you get to hear about it pretty fast. The second scenario, losing to the competiton, happens slowly over months or years, and by the time you slip into obscurity so many events have occurred it’s hard to pinpoint exactly where you went wrong.

Urgency is no bad thing, then, but I continue to be bothered by its dominance in web development. Too much urgency leads to a loss of quality and a reduction of strategic actions. Why does this happen in web development more than elsewhere? Here are some thoughts…

1. Updates are cheaper on the web

Rolling out a new version of a website onto a server shouldn’t be regarded as a trivial affair, but it’s a lot easier than other means of releasing software. So it’s easier to release a slightly rushed product if we know the only copies anywhere in the world live on servers which are entirely under our control.

Slightly more costly than updating our own servers is the job of the IT department which has to update an application on everyone’s desktop. Here, not only do you have to be sure the desktop machines are all switched on and ready to receive the software, but an error for one person could mean they cannot work at all the next day. More costly than that is the job of burning, boxing, and shipping a DVD to hundreds of retail stores.

For both those cases, and for many other ways of releasing software, the cost of repetition and the cost shipping something substandard is very high. But in many web applications, where the cost of distribution and the cost of error is relatively low, the consequences of problems that come from an urgent release are much less costly.

2. Problems hurt less on the web

As users, our relationship with websites and web applications tends to be less engaged than with most other applications. So although we all know undue urgency leads to more faults being released, it’s perhaps more forgivable on the web.

If I’m using a website and I encounter an error, I know the problem — the website — is theirs. I can come back later or try an alternative (there are always plenty of alternatives on the web). Either way, I’m probably not going to get very upset.

By contrast, if I encounter a problem with software running on my machine then any error is going to hurt me much more: that’s a problem with my software… it’s causing my machine to go wrong… I went to the trouble of installing that software and now it’s giving me problems… this is not good. Even if the software was installed by my IT department it’s still my software, and it’s core part of my daily work — finding a workaround is going to be difficult.

3. Competition is more visible on the web

One of the drivers of urgency is beating the competition, and if you and your competitors all work on the web, then they are likely to make (cheap) incremental releases, and you will notice almost every one.

If competitor A releases a new feature in week 1 then you’ll notice and ask how you can do that yourselves. If competitor B releases another new feature in week 2 then you’ll notice it and ask how you can beat that, too. By the time competitor C releases a third thing in week 3 you’ve got the pressure of the three things being delivered by your competition and nothing to show yourself.

Things are different as you move offline. Consider software companies in a competitve marketplace that see software ship features in batches. When your competition ships their new release with new features you’ll be able to assess all those features as a whole. Of course there will still be intense pressure to deliver, but your view will be more rounded and decisions of importance will play a greater role than decisions of urgency.

Consider also the internal team which delivers software which is internally-facing. Your life in this team will be no less demanding than in any other, but one thing you’ll be spared is much pressure from competitive teams. If you’re delivering the next version of the payroll system then there will be pressure of deadlines, there will be pressure of budget, but you’re very unlikely to get an unexpected new requirement due to the payroll software team at your competitor suddenly releasing their new e-mail alert module.

None of this is to say that competition is more fierce on the web. That may or may not be the case, but it’s not what I’m interested in here. The point is that the competition seems more fierce and is much more visible. This encourages urgency.

4. Boundaries are blurred on the web

A wordprocessor helps you write documents, a media player plays audio and video, but a website… a website doesn’t have clear boundaries.

Although I wrote above of competition, exactly who is a competitor on the web is not always certain. I once worked with an e-commerce music retailer who suddenly decided they needed to provide a web mail service for their customers. I’ve worked with a law firm who wanted to turn their brochureware site into a recruitment platform. Yahoo! evolved from a directory into a portal, embracing almost everything they could while the world struggled to understand what a portal might not be. Meanwhile, stepping away from the web, I don’t know of a spreadsheet which carries a discussion forum or an HR system which recommends movies.

While all this clearly accounts for the quantity of demands that occur in web development, it also accounts for the urgency because that steady drip-drip-drip of features that comes from our competition also comes from those who aren’t direct competition but are just out there on the web. The drip-drip-drip becomes a trickle (and maybe even a flood) pretty quickly, and the pressure to deliver new things is even greater.

5. It’s easier to have more stakeholders on the web

Because the web means so many things to so many people — contributing to those blurred boundaries — many more people within an organisation will consider themselves stakeholders. Marketing people see it as a marketing tool, the recruitment team see it as a recruitment tool, the sales team see it as a sales tool, and so on. And of course each one of them is right: it can be all these things.

With so many diverse stakeholders, we can expect so many more pressing deadlines, and hence much more urgency. More stakeholders also means communication and prioritisation is more difficult. It’s difficult to get everyone into the same room, it’s difficult to bring everyone to the same point of understanding, it’s difficult to weigh completely different kinds of requirements against each other. It doesn’t matter that everyone is working for the same company and therefore, at a higher level, the same goal. That goal is usually at too high a level to make a difference to everyone’s day-to-day needs.

The least pleasant example of this I remember is a colleague at a previous company who was the project manager for one of our clients. Although he was their single point of contact they managed to have several single points of contact, none of whom were very good at speaking to the others. He found he had responsibilities to the marketing manager, the distribution manager and the IT director. They all had pressing demands, and he had no contact with anyone who could bring them all into line. Indeed, that seemed to be by design, as we suspected the client used this as a way of squeezing us as a supplier. The relationship did not end happily. Although that’s a very extreme example it does highlight the kind of difficulties of having many stakeholders.

A final word

Cheaper updates, less damaging problems, more visible competition, blurred boundaries and more stakeholders. This tends to be the world of web development, and therefore a world in which urgency plays a more dominant role.

Yet we can think of web development projects in which importance dominates urgency. One which springs to mind is an online banking site: this is web development if ever anything is, but one in which doing things quickly will always be trumped by doing things properly (whatever “properly” may mean for that bank). But this is also an environment where many of the criteria I’ve listed don’t hold. Most obviously updates will be fairly costly, any problems will be very damaging, and the boundary of what the site does is very well defined.

So the criteria I’ve set out above are more to explain than define. Web development tends to take place in an environment that’s different from many other software development environments, and I think it’s those features of that environment that I’ve listed that explain why urgency predominates.

I said at the beginning this vexes me because urgency causes visible problems, and eliminating urgency would seem to help eliminate those problems. But I also said I’m usually dissatisfied with one unmet ideal or another. I’ve also worked in environments where there is a distinct lack of urgency. That, of course, leads to waste and stagnation. And that’s really, really, vexing…

Staggering though it may be, it turns out that people are different. It also turns out that certain kinds of people are different to other kinds of people. And a corollary of this is that people who aren’t software engineers tend to have a different perspective to those who are.

For example, I spend a lot of my time trying to maximise our efficiency. Maybe this is because I’m a techie kind of person; I see the same motivation among the project managers, developers, and others in our software team. But very often we face the prospect of having to start a piece of work without having all the details available to finish the job. Maybe we’re waiting for information from a third party, maybe the visual design isn’t complete, maybe a decision still needs to be made by people elsewhere in the organisation. Whatever it is, we’re faced with the prospect of starting a piece of work knowing that we may not be able to complete it without an interruption.

This is clearly going to be frustrating to my techie, left-brained approach to life. It undermines my efforts to be efficient, organised, anally retentive, and generally less fun at parties.

I know it would frustrate others, too. Agile advocate Simon Baker recently railled against organisations that didn’t make product owners sufficiently available to provide the relevant feedback and information. He wrote

If the project is vital to the business, then the company can always find a way to provide a full-time and colocated Product Owner. If they say they can’t, it really means they won’t. Quite simply, they’re not prepared to do what is necessary to achieve it, and frankly, if they’re not going to take the project seriously why should you?

Ouch. I’ve felt the pain that Simon felt when he wrote those words, but we shouldn’t rush to make harsh judgements on business experts who are facing pressures of their own.

We software people can talk all we want about our methodologies, but sometimes we need to wake up to the cold, wet slap of reality. The fact is software development is not the be-all and end-all of most businesses — far less is efficient software development methodologies. Sometimes it’s more important to be working on the most important thing inefficiently than it is to be working on the second-most important thing efficiently.

At Guardian Unlimited we motivate ourselves by measuring velocity — the number of units of work we complete in a given period. But if we’re not careful we can focus on that too much and be in danger of missing the bigger picture.

Some time ago I was speaking to one of our internal customers, explaining why developers pushed back on incomplete specifications, and the motivation of being efficient and achieving target velocity. “I don’t give a stuff about velocity,” she said, “I just want the thing built.” The point is well taken. Sometimes we need to remember what the word “customer” means.

Compare and contrast two blog entries that popped up in my RSS reader on the same day last week. In the blue corner Joel O’Software, fighting for private offices. And in the red corner, Martin Fowler, battling it out for continuous collaboration between developers and their customers.

Naturally, they’re not really at opposite ends of a spectrum — in fact, their posts are about different things — and they’ll find a lot to agree with each other. But look at a select part of each of their text and see the contrast. Here’s Joel:

Not every programmer in the world wants to work in a private office. In fact quite a few would tell you unequivocally that they prefer the camaradarie and easy information sharing of an open space. Don’t fall for it. They also want M&Ms for breakfast and a pony.

And here’s Martin:

One of Kent’s suggested names for ‘Agile’ was conversational software development – the point being that it’s a two way communication. This isn’t something like a telecoms protocol that you can define, but the back and forth discussions about how software can enhance the business are where the real value lives. Much of this conversation is of half-baked ideas, some of which grow into valuable features – often ones that aren’t things that the customer originally thought of.

It’s notable how two people renowned for being leaders in software can be veering apart on what should be a fundamental issue: how should people interact? Joel is for concentration, Martin is for conversation.

It’s notable also that they do different things in the software world. Joel produces shrink-wrapped products on behalf of his own company. So does Microsoft, the company Joel used to work for and who he praises for “putting literally everyone in individual, private offices”. Martin is a gun for hire (via his employer), called in to consult on a variety of projects for different companies, no doubt 90% of the time producing in-house software for each client.

I can’t escape the feeling that their respective backgrounds inform their respective views, though wouldn’t for a second think that implies that one approach always suits one kind of output.

So, should one prefer concentration or conversation? Obviously[*] it depends on several factors, and here’s the way I see it…

First, it must come from whoever you start with. If Joel O’Software starts a one-man band and likes to work in silence, then takes on his first employee, he’s not going to want his hire to keep piping up with questions every two minutes. Similarly, if you’ve created your business by extolling the virtues of pair programming then you’ll be looking exclusively for developers who will continue that good work. If you start with one introvert or extravert, you’ll grow from there.

Second, it depends on where you think your strengths are as an organisation. Joel talks a lot about productivity and algorithms, while Martin tends to talk of people and methodologies. Each also has an interest in the other’s topics, but their chosen hot subjects are where they see the biggest gaps, and where they think they can most make a difference.

Third, it’s about how you see your team. I suspect Joel has very low staff turnover, hires developers very infrequently, and there’s no doubt he puts a lot of effort into picking the cream of the crop: he’s in a buyer’s market, and his developers will all be very smart. Martin will inevitably work with a much broader range of companies. While they will of course have made a very smart decision to hire him and his colleagues (ahem) they will tend more towards the market norm, and will also tend to be fairly large development teams — even if individual project teams are smaller. Thus Martin is going to be much more concerned about sharing information between developers, evolving designs collaboratively, establishing standards and keeping those standards refreshed.

Finally, it’s about how you see the long term. Again, low staff turnover and a tight-knit team means Joel is less concerned about knowledge silos, but an average corporate team will have average turnover and will have its average share of crises. Knowledge sharing and reducing single points of failure is essential in these cases.

All of that is why I favour conversational development. Knowledge sharing and evolving ideas is key to me as a general principle, all the more so that Guardian Unlimited is such a diverse site that there’s just so much to know. That doesn’t mean it’s easy for everyone, but for me a typical team will be stronger if it keeps every last bit of information flowing round, ideas constantly exchanged and checked, and experience continuously refreshed and revised. Concentration is often needed, but too often the price paid is too high, and is only found when one person is seconded to another project, has left, or is holiday.

[*] I think there’s a progression whenever you ask “Is A or B better?” Naive or inexperienced people will always pick one or the other. Eventually they’ll come to hedge their bets because they realise things are more subtle than they previously thought, or else it just makes them sound wiser. Finally they may reach a point when they are (or regard themselves as) leaders in their field and act as evangelists or iconoclasts pushing one or other opinion heavily. You’ll see a lot of hedging on this blog.