Category: Working practices

QCon London 2008: A Michelin-starred deli

There were very few moments for me during QCon London 2008 of earth-shaking enlightenment — if any. But every hour of the three days of the conference there were insights and guidance that could be tucked away, and reused later to save hours, days or weeks of time elsewhere. Snake-oil salesmen where thin on the ground, and instead there were dozens of people saying one or both of:

  • This is what we did; and
  • This is what you can do.

No magic, no silver bullets, but plenty of solid advice and experience.

A good example of both of these was Randy Shoup of eBay. He had nothing to sell (other than the good name of eBay, perhaps) and his presentation was very clearly constructed to show their principles of scalability, and some concrete examples of how these work in practice. You probably wouldn’t use their periodic batch processing method to generate recommendations — if only because it’s odds on you don’t have a recommendation system — but you could take the overarching principle of “async everywhere” and apply that to the next scalable application that you need to work on.

Even the very specific presentations contained valuable points that could be generalised and reused. For example, Matt Youill and Asher Glynn of Betfair talked through how they scaled the transaction processing on their servers by a hundred-fold. Guardian.co.uk doesn’t need that kind of throughput, so the details were primarily of intellectual benefit. But a key practical lesson was how they approached the problem: by presenting it to industry players as a challenge carrying great kudos to the winning company.

All of this was summed up very nicely by the team from BBC News: John O’Donovan, Kevin Hinde and Ross Heritage. They were asked how they managed performance testing for the iPlayer. John spent a few moments describing some of the techniques they employed, but got to the point when he realised the audience really wanted some eye-opening enlightenment which he didn’t have. At this moment Kevin stepped forward and said straight out “There’s no secret sauce”. Indeed not: they just work hard and stick to strong principles.

QCon offered little in the way of secret sauces, but it did contain dozens and dozens of great ingredients you could take away and use to concoct your own wonderful dishes.

And with that analogy pushed to breaking point, I think we should leave it there.

The unstoppable urgency of web development

While I’m usually proud of the work I’m involved with, I’m rarely happy for long. There are always ways to improve, and I’m usually dissatisfied by one unmet ideal or another. Almost since I started in this field I’ve been vexed by how much of web development is “urgent” rather than “important”. This is not merely of theoretical interest; it vexes me because dealing with important issues gets you to something of lasting and strategic value, while dealing with urgent issues gets you to the next day. It’s that unmet ideal of achieving more long-lasting value that causes me to think about this.

I want to give some context around the claim that there’s more urgency in web development, and then I want to offer some ideas about why this might be. If you want the brief version, I think what makes the web generate such urgency is that…

  1. Updates are cheaper;
  2. Problems hurt less;
  3. Competition is more visible;
  4. Boundaries are blurred; and
  5. It’s easier to have more stakeholders.

But first, some context.

Picture will go here when I have timeSome context

Although I do think web development has a surfeit of urgency, it’s fair to say that “web development” is not a clearly defined area. So much software today has some link or other to the web, it’s usually unfair to say definitively that something is or is not web development. So it’s a matter of degrees, and looking back on the software development I’ve been involved with I’d say the more web-oriented something has been the more I’ve had to deal with urgent issues, and the less web-oriented something has been the more I’ve had to deal with important issues. The most urgency comes when working on websites, and the least when working on applications which may or may not use the web for a bit of minor communication.

This issue of urgency also seems to be the perception and expectation of non-technical people. This is how we get the phrase “Internet time” and why for years I’ve heard people saying “but this is the Internet” as a justification for tight deadlines. This is something that continues today, even after the unsustainable boom and bust of the dot.com years.

When urgency is too pressing we risk doing a bad job, and releasing something which isn’t up to scratch. Of course, when urgency isn’t pressing enough we lose ground to our competitors. And of those two scenarios it’s the first one, doing a bad job, which is easiest to spot: a bug bites someone on the nose and you get to hear about it pretty fast. The second scenario, losing to the competiton, happens slowly over months or years, and by the time you slip into obscurity so many events have occurred it’s hard to pinpoint exactly where you went wrong.

Urgency is no bad thing, then, but I continue to be bothered by its dominance in web development. Too much urgency leads to a loss of quality and a reduction of strategic actions. Why does this happen in web development more than elsewhere? Here are some thoughts…

1. Updates are cheaper on the web

Rolling out a new version of a website onto a server shouldn’t be regarded as a trivial affair, but it’s a lot easier than other means of releasing software. So it’s easier to release a slightly rushed product if we know the only copies anywhere in the world live on servers which are entirely under our control.

Slightly more costly than updating our own servers is the job of the IT department which has to update an application on everyone’s desktop. Here, not only do you have to be sure the desktop machines are all switched on and ready to receive the software, but an error for one person could mean they cannot work at all the next day. More costly than that is the job of burning, boxing, and shipping a DVD to hundreds of retail stores.

For both those cases, and for many other ways of releasing software, the cost of repetition and the cost shipping something substandard is very high. But in many web applications, where the cost of distribution and the cost of error is relatively low, the consequences of problems that come from an urgent release are much less costly.

2. Problems hurt less on the web

As users, our relationship with websites and web applications tends to be less engaged than with most other applications. So although we all know undue urgency leads to more faults being released, it’s perhaps more forgivable on the web.

If I’m using a website and I encounter an error, I know the problem — the website — is theirs. I can come back later or try an alternative (there are always plenty of alternatives on the web). Either way, I’m probably not going to get very upset.

By contrast, if I encounter a problem with software running on my machine then any error is going to hurt me much more: that’s a problem with my software… it’s causing my machine to go wrong… I went to the trouble of installing that software and now it’s giving me problems… this is not good. Even if the software was installed by my IT department it’s still my software, and it’s core part of my daily work — finding a workaround is going to be difficult.

3. Competition is more visible on the web

One of the drivers of urgency is beating the competition, and if you and your competitors all work on the web, then they are likely to make (cheap) incremental releases, and you will notice almost every one.

If competitor A releases a new feature in week 1 then you’ll notice and ask how you can do that yourselves. If competitor B releases another new feature in week 2 then you’ll notice it and ask how you can beat that, too. By the time competitor C releases a third thing in week 3 you’ve got the pressure of the three things being delivered by your competition and nothing to show yourself.

Things are different as you move offline. Consider software companies in a competitve marketplace that see software ship features in batches. When your competition ships their new release with new features you’ll be able to assess all those features as a whole. Of course there will still be intense pressure to deliver, but your view will be more rounded and decisions of importance will play a greater role than decisions of urgency.

Consider also the internal team which delivers software which is internally-facing. Your life in this team will be no less demanding than in any other, but one thing you’ll be spared is much pressure from competitive teams. If you’re delivering the next version of the payroll system then there will be pressure of deadlines, there will be pressure of budget, but you’re very unlikely to get an unexpected new requirement due to the payroll software team at your competitor suddenly releasing their new e-mail alert module.

None of this is to say that competition is more fierce on the web. That may or may not be the case, but it’s not what I’m interested in here. The point is that the competition seems more fierce and is much more visible. This encourages urgency.

4. Boundaries are blurred on the web

A wordprocessor helps you write documents, a media player plays audio and video, but a website… a website doesn’t have clear boundaries.

Although I wrote above of competition, exactly who is a competitor on the web is not always certain. I once worked with an e-commerce music retailer who suddenly decided they needed to provide a web mail service for their customers. I’ve worked with a law firm who wanted to turn their brochureware site into a recruitment platform. Yahoo! evolved from a directory into a portal, embracing almost everything they could while the world struggled to understand what a portal might not be. Meanwhile, stepping away from the web, I don’t know of a spreadsheet which carries a discussion forum or an HR system which recommends movies.

While all this clearly accounts for the quantity of demands that occur in web development, it also accounts for the urgency because that steady drip-drip-drip of features that comes from our competition also comes from those who aren’t direct competition but are just out there on the web. The drip-drip-drip becomes a trickle (and maybe even a flood) pretty quickly, and the pressure to deliver new things is even greater.

5. It’s easier to have more stakeholders on the web

Because the web means so many things to so many people — contributing to those blurred boundaries — many more people within an organisation will consider themselves stakeholders. Marketing people see it as a marketing tool, the recruitment team see it as a recruitment tool, the sales team see it as a sales tool, and so on. And of course each one of them is right: it can be all these things.

With so many diverse stakeholders, we can expect so many more pressing deadlines, and hence much more urgency. More stakeholders also means communication and prioritisation is more difficult. It’s difficult to get everyone into the same room, it’s difficult to bring everyone to the same point of understanding, it’s difficult to weigh completely different kinds of requirements against each other. It doesn’t matter that everyone is working for the same company and therefore, at a higher level, the same goal. That goal is usually at too high a level to make a difference to everyone’s day-to-day needs.

The least pleasant example of this I remember is a colleague at a previous company who was the project manager for one of our clients. Although he was their single point of contact they managed to have several single points of contact, none of whom were very good at speaking to the others. He found he had responsibilities to the marketing manager, the distribution manager and the IT director. They all had pressing demands, and he had no contact with anyone who could bring them all into line. Indeed, that seemed to be by design, as we suspected the client used this as a way of squeezing us as a supplier. The relationship did not end happily. Although that’s a very extreme example it does highlight the kind of difficulties of having many stakeholders.

A final word

Cheaper updates, less damaging problems, more visible competition, blurred boundaries and more stakeholders. This tends to be the world of web development, and therefore a world in which urgency plays a more dominant role.

Yet we can think of web development projects in which importance dominates urgency. One which springs to mind is an online banking site: this is web development if ever anything is, but one in which doing things quickly will always be trumped by doing things properly (whatever “properly” may mean for that bank). But this is also an environment where many of the criteria I’ve listed don’t hold. Most obviously updates will be fairly costly, any problems will be very damaging, and the boundary of what the site does is very well defined.

So the criteria I’ve set out above are more to explain than define. Web development tends to take place in an environment that’s different from many other software development environments, and I think it’s those features of that environment that I’ve listed that explain why urgency predominates.

I said at the beginning this vexes me because urgency causes visible problems, and eliminating urgency would seem to help eliminate those problems. But I also said I’m usually dissatisfied with one unmet ideal or another. I’ve also worked in environments where there is a distinct lack of urgency. That, of course, leads to waste and stagnation. And that’s really, really, vexing…

Amazingly, some people aren’t motivated by efficiency

Staggering though it may be, it turns out that people are different. It also turns out that certain kinds of people are different to other kinds of people. And a corollary of this is that people who aren’t software engineers tend to have a different perspective to those who are.

For example, I spend a lot of my time trying to maximise our efficiency. Maybe this is because I’m a techie kind of person; I see the same motivation among the project managers, developers, and others in our software team. But very often we face the prospect of having to start a piece of work without having all the details available to finish the job. Maybe we’re waiting for information from a third party, maybe the visual design isn’t complete, maybe a decision still needs to be made by people elsewhere in the organisation. Whatever it is, we’re faced with the prospect of starting a piece of work knowing that we may not be able to complete it without an interruption.

We built the car very efficiently...This is clearly going to be frustrating to my techie, left-brained approach to life. It undermines my efforts to be efficient, organised, anally retentive, and generally less fun at parties.

I know it would frustrate others, too. Agile advocate Simon Baker recently railled against organisations that didn’t make product owners sufficiently available to provide the relevant feedback and information. He wrote

If the project is vital to the business, then the company can always find a way to provide a full-time and colocated Product Owner. If they say they can’t, it really means they won’t. Quite simply, they’re not prepared to do what is necessary to achieve it, and frankly, if they’re not going to take the project seriously why should you?

Ouch. I’ve felt the pain that Simon felt when he wrote those words, but we shouldn’t rush to make harsh judgements on business experts who are facing pressures of their own.

We software people can talk all we want about our methodologies, but sometimes we need to wake up to the cold, wet slap of reality. The fact is software development is not the be-all and end-all of most businesses — far less is efficient software development methodologies. Sometimes it’s more important to be working on the most important thing inefficiently than it is to be working on the second-most important thing efficiently.

At Guardian Unlimited we motivate ourselves by measuring velocity — the number of units of work we complete in a given period. But if we’re not careful we can focus on that too much and be in danger of missing the bigger picture.

Some time ago I was speaking to one of our internal customers, explaining why developers pushed back on incomplete specifications, and the motivation of being efficient and achieving target velocity. “I don’t give a stuff about velocity,” she said, “I just want the thing built.” The point is well taken. Sometimes we need to remember what the word “customer” means.

Conversation versus concentration

Compare and contrast two blog entries that popped up in my RSS reader on the same day last week. In the blue corner Joel O’Software, fighting for private offices. And in the red corner, Martin Fowler, battling it out for continuous collaboration between developers and their customers.

Naturally, they’re not really at opposite ends of a spectrum — in fact, their posts are about different things — and they’ll find a lot to agree with each other. But look at a select part of each of their text and see the contrast. Here’s Joel:

Not every programmer in the world wants to work in a private office. In fact quite a few would tell you unequivocally that they prefer the camaradarie and easy information sharing of an open space. Don’t fall for it. They also want M&Ms for breakfast and a pony.

And here’s Martin:

One of Kent’s suggested names for ‘Agile’ was conversational software development – the point being that it’s a two way communication. This isn’t something like a telecoms protocol that you can define, but the back and forth discussions about how software can enhance the business are where the real value lives. Much of this conversation is of half-baked ideas, some of which grow into valuable features – often ones that aren’t things that the customer originally thought of.

It’s notable how two people renowned for being leaders in software can be veering apart on what should be a fundamental issue: how should people interact? Joel is for concentration, Martin is for conversation.

It’s notable also that they do different things in the software world. Joel produces shrink-wrapped products on behalf of his own company. So does Microsoft, the company Joel used to work for and who he praises for “putting literally everyone in individual, private offices”. Martin is a gun for hire (via his employer), called in to consult on a variety of projects for different companies, no doubt 90% of the time producing in-house software for each client.

I can’t escape the feeling that their respective backgrounds inform their respective views, though wouldn’t for a second think that implies that one approach always suits one kind of output.

So, should one prefer concentration or conversation? Obviously[*] it depends on several factors, and here’s the way I see it…

First, it must come from whoever you start with. If Joel O’Software starts a one-man band and likes to work in silence, then takes on his first employee, he’s not going to want his hire to keep piping up with questions every two minutes. Similarly, if you’ve created your business by extolling the virtues of pair programming then you’ll be looking exclusively for developers who will continue that good work. If you start with one introvert or extravert, you’ll grow from there.

Second, it depends on where you think your strengths are as an organisation. Joel talks a lot about productivity and algorithms, while Martin tends to talk of people and methodologies. Each also has an interest in the other’s topics, but their chosen hot subjects are where they see the biggest gaps, and where they think they can most make a difference.

Third, it’s about how you see your team. I suspect Joel has very low staff turnover, hires developers very infrequently, and there’s no doubt he puts a lot of effort into picking the cream of the crop: he’s in a buyer’s market, and his developers will all be very smart. Martin will inevitably work with a much broader range of companies. While they will of course have made a very smart decision to hire him and his colleagues (ahem) they will tend more towards the market norm, and will also tend to be fairly large development teams — even if individual project teams are smaller. Thus Martin is going to be much more concerned about sharing information between developers, evolving designs collaboratively, establishing standards and keeping those standards refreshed.

Finally, it’s about how you see the long term. Again, low staff turnover and a tight-knit team means Joel is less concerned about knowledge silos, but an average corporate team will have average turnover and will have its average share of crises. Knowledge sharing and reducing single points of failure is essential in these cases.

All of that is why I favour conversational development. Knowledge sharing and evolving ideas is key to me as a general principle, all the more so that Guardian Unlimited is such a diverse site that there’s just so much to know. That doesn’t mean it’s easy for everyone, but for me a typical team will be stronger if it keeps every last bit of information flowing round, ideas constantly exchanged and checked, and experience continuously refreshed and revised. Concentration is often needed, but too often the price paid is too high, and is only found when one person is seconded to another project, has left, or is holiday.

[*] I think there’s a progression whenever you ask “Is A or B better?” Naive or inexperienced people will always pick one or the other. Eventually they’ll come to hedge their bets because they realise things are more subtle than they previously thought, or else it just makes them sound wiser. Finally they may reach a point when they are (or regard themselves as) leaders in their field and act as evangelists or iconoclasts pushing one or other opinion heavily. You’ll see a lot of hedging on this blog.