Obscure management phrases for real people: No. 3, Governance

·

This is the third and final installment of a series attempting to redeem words tarnished by management. Previously I looked at “strategic” and “leadership”. Today: governance.

This is a word that doesn’t tend to get used much among software development teams, and when it does it tends to carry the claustrophobic feel of heavy bureaucracy: ITIL, ISO, and lots and lots of very dull paperwork. But while it might not be thrill-a-minute stuff it’s something that the software team on the ground does need to be aware of: not just what it is, but how it works in your organisation.

Paperwork is not proportional to trustAbout governance

Governance is about providing the link between what people do in the software team and what people in senior management need and expect. Without that link there would be no mutual trust. That trust is important to the tech teams because, to be frank, none of us would be here without senior managers’ say-so. They provide the time and resources for us to do our jobs — not just the physical working environment and pay-cheques, but a meaningful and stable programme of work.

Here are a couple of examples.

Example 1: Life with poor governance

The importance of governance came home to me a few years ago when I worked on a project which involved a group of management execs somewhat separate from the development team. As with many such projects there were all sorts of unexpected problems, but we on the development team had the great benefit of working closely with the end users who who provided regular guidance and advice. As a result we developed something which more or less made those end users very happy — they didn’t get everything they wanted but they were happy that they got a good system within the bounds of budget and time.

The management execs, meanwhile, met regularly to track our budget, which was used responsibly and didn’t provide too many alerts.

Unfortunately when the final product was delivered the executive group was distinctly unhappy. It turns out they had expected one or two key features which the end users had decided they could cut among some of the compromises they had to make. When the development team and end users were proud of their finished work the executive group were disappointed.

The result was that future work with this group was incredibly difficult, because trust had been lost — they authorised future work only after a great deal of pain, and we generally had a work an awful lot harder to get anything done.

As result of poor governance the people on the ground had a much harder time in their day-to-day lives.

It’s clear to me that no one person or group was at fault here, but as a development team we could have done better. We should have flagged up not just budget changes but also changes in the evolving feature set.

Example 2: Governance on the guardian.co.uk

Good governance isn’t about reporting absolutely everything upwards, because you want your executive group to focus on the relevant issues.

On the recently completed guardian.co.uk rebuild project we were careful about what did and did not go to the board. For example, while the development team estimated and costed work in “ideal days” or “story points” we were careful to translate these into pounds sterling for the board. Ideal days is a useful abstraction for development estimation, but a pointless additional layer for a board.

In terms of the evolving feature set, change was tracked carefully within the development team and percolated upwards. But again, the information wasn’t conveyed verbatim, because development teams worked on lower-level tasks than were meaningful to users. So all lower-level task changes were translated into higher-level feature changes when presented to the board.

The result of this good governance was that when the the project ended both the board and the development team were united in recognising a job well done. We have further plans now, and although we’ll still have to justify each of them we are at least starting from a position of trust.

By the way, all this governance this did involve paperwork — in the form of writing board reports. But that’s only because paper is often a good vehicle for communication, and communication is central to building trust. We shouldn’t dismiss governance simply because it involves careful communication.

Governance in review

Good governance is therefore all about trust through communication; for software teams it means they get the trust and support of those senior people who shape their working environment and daily lives. Without good governance that trust won’t be forthcoming. Understanding how it works in your organisation means you have a route to ensuring you have senior management support when you need it.

Obscure management phrases for real people: No. 2, Leadership

·

This is the second in a short series of articles intended to restore the reputation of certain words — words that have been abused by managers and therefore have likely become meaningless to people who have to listen to them. Today: “leadership”.

“Leadership” is surely a word that needs saving from the management jargon-fest. It does mean something real and important, but I think it’s too easy to switch off when you hear it. It’s so loaded with portent that it’s easy to mistake the speaker as deluded or (at best) mistaken — after all, how many Winston Churchills, Nelson Mandelas or Martin Luther Kings can you expect to find in your organisation? Also, unless you recognise a revolution in progress, it’s hard to spot leadership at work.

Yet leadership is important and possible in even the smallest groups or organisations. As with the discussion of “strategic”, earlier, this article is part apology, part explanation.

Leadership in the animal kingdomAbout “leadership”

There is much that’s been written about leadership, but it tends to be for ambitious management-types [1, 2, 3], and therefore ignorable by the people who probably feel they’re doing the real work. Yet it’s important for everyone.

For me, leadership is about encouraging groups of people to being doing things they wouldn’t do if left to their own devices. These might be people inside or outside your team.

So if your team is stuck in a rut, then someone needs to exercise some leadership to help you out of it. Or perhaps there’s a dysfunctional relationship between your department and the rest of your company; again, it requires someone with some leadership to change the way people work together.

These situations are not just about implementing changes, they’re about changing the otherwise-routine behaviours of groups of people. And leadership doesn’t need to come only from those with the keys to the executive washroom. Here are some examples..

Example: Increasing the value of our product

This story comes from one of my former companies, and is about someone I’ll call “Mike”, because that was his name.

Our company produced shrink-wrapped software for search engine marketing, and its most common function was to produce search-engine-friendly content for sites which didn’t already have it. (An aside here: that kind of thing isn’t too respectable today, so let me assure you of three things: (1) this was a very long time ago, when the rules we know today weren’t written, (2) we always made sure our clients used the software for respectable purposes, and (3) the company today is quite a different beast and enjoys strong partnership relations with all the major search engines.) Mike was one of the technical consultants for the sales staff: he’d go along with them to sales meetings, answer technical questions, feed back ideas to the developers, configure installations, and so on.

The core module of the software was given manually-entered information about your company (company name, key product names, common related terms) and generated the content. This was a nice, simple sales proposition and was relatively easy to sell: we put you on search engines — simple, eh? However, we always felt it was undervalued, because it took a lot of time and effort to demonstrate and explain, and it was incredibly difficult to justify anything more than quite a low price on something as vague as “visibility”.

There was another module, too, which mined your company’s product database and drew out terms from that so you didn’t have to key them in yourself. If you’ve got 5,000 products and you were actually going to key them all in, that’s a real time-saver. But database connectivity was quite a new and and cumbersome thing (I told you this was a long time ago), difficult for the sales people to understand, and difficult to convince clients that they should spend time doing this when the only real value was to save you some typing which you probably weren’t going to do anyway. In the end our software was a technically advanced product of lowish-value and with a fancy database add-on.

But one day Mike started seeing things differently. He realised that by mining a product database the system was raising the profile of every single product that your company offered. Each of those products was a direct revenue stream for the company, and by raising the visibility of those products the system was going to put those products in front of more people, leading to a proportional rise is sales. Suddenly you could put numbers into a spreadsheet, and even a very conservative rise in sales would mean our system could be incredibly valuable to companies. We could comfortably value our software much more highly — as a proportion of your company’s annual turnover.

Mike went to our CEO with his new perspective, and our CEO was enthusiastic. He created a PowerPoint presentation, and took it round the sales people and his fellow technical consultants. It was clear to us that we had seriously misunderstood our own work. We had a new sense of excitement — we were sitting on something really valuable and hadn’t realised it. Nothing physical had changed, but we had a new outlook. We could explain what we did in a new and much more compelling way. We started using the phrase “return on investment” with conviction, and it actually meant something to our clients.

It wasn’t instantaneous, of course. Mike went out to trial our new outlook and our new presentations in front of clients. Lessons were learnt — the good and the bad — and were fed back to the team. Occasionally there was a confusion or misunderstanding, or subtle presentational tweaks needed to be made so as to make our message clear. Mike would ensure we kept learning the lessons from each other, and that our understanding was common and thorough.

I think the first actual sale of our software with this new perspective was ten times the previous average, and four or five times higher than anything we had sold before. And everyone was a winner: our clients had a tool to increase their revenue, and we were making more from our product.

Mike might have been one of the many people in the team, but he showed leadership. He changed the outlook and the behaviour of a large group of people. Nothing changed physically: we didn’t change our software, we didn’t change our staff. But we changed our attitude, we changed the way we talked about ourselves and our product, and we changed the benefits we offered our clients.

Oh yes, and we changed our revenue. That went up.

More leadership examples

There are numerous other examples of leadership that I’ve witnessed, but I fear writing about the protagonists as if they were saints, and I don’t want to embarrass them. So instead I’ll list a few examples only briefly…

  • The developer who lead the replacement of our proprietary application server, bringing in an open source (and much more managable) alternative. While everyone agreed it was important to do, he designed the architecture, sketched out the plan, and guided the other developers through the implementation. We still had a project manager and many others involved, and I can still say it was a team effort, but that developer saw it through from beginning to end and was a pivotal authority throughout.
  • The sales manager who led his disparate transatlantic sales people to be much more accountable and transparent about their prospective clients. He created a system of gates, in which the sales people needed to categorise their prospects, and ensured each member of the team shared their progress in a weekly conference call. With increased accountability, the sales team became much more trusted and respected throughout the organisation.
  • The team lead who helped us integrate a third-party Javascript product. He engineered an appropriate test framework, coached his team on its appropriate use, oversaw the development work, and volunteered himself as an out-of-hours technical point of contact in case of any problems. Something that was previously deemed risky and made people nervous was ultimately held up as example of how people might do things on future projects.
  • The business development manager who translated Agile planning techniques into his own (non-technical) department. He used these to regularly organise and prioritise a demand pipeline that was not only impossibly large, but was so diverse the various demands weren’t easily comparable. He ensured the various stakeholders discussed their needs together and came to a common agreement, allowing the department to regularly present a clear, consistent message to the rest of the business.

I can be quite certain that when these people did these things they didn’t for one second think they were demonstrating leadership. The sales manager will have thought he was just implementing good practice; the app-server developer will have thought he was just sorting out a long-standing problem. And in all cases they’d be right. But they demonstrated leadership, too.

Leadership in review

I’ve deliberately picked out people who were in the middle — not the top — of their respective organisations. The individual things they did towards each achievement ran over a sustained period, and were interspersed throughout their day-to-day jobs. But at the end of this they had changed the way their colleagues worked, and they were all undoubtedly better teams as a result.

So although leadership might sound overly-grand when referenced by management-types, it really can come from anywhere. I hope these examples show that leadership is a real and important quality, needed at all levels and in every organisation.

Obscure management phrases for real people: No. 1, Strategic

· 2 Comments ·

This is the first in a haphazard series which takes various management phrases and tries to ground them in reality. First, though, a bit about why I’m writing this.

A poor choice of words can be damaging. There are numerous times I’ve received an e-mail introduction from a company, and after a couple of paragraphs of impressive-sounding buzzwordology I’ve thought to myself “Yes, but what do you actually do?” Sometimes, though, some of those words actually translate into a tangible reality, and mean something for the lives of real people — even those who don’t spend their days thumping boardroom tables.

Sometimes, also, I find myself using management jargon in front of management-types, and then repeating it in front of non-management-types, which is probably not very helpful to the dialogue. But while these words may sound empty or obtuse, they do relate to things that impact on real people’s lives, and that’s why I use them.

So if I, or someone else, has used such jargon in front of you then please don’t shut down immediately. The consequences of these words affect our working day.

Today’s word is… strategic.

About “strategic”

I think the word strategic is more useful than the word strategy, because something can be strategic without you actually having a strategy. If a strategy is the route you want to go down, then strategic refers to the general direction you want to go in. You can know which direction you want to go in without knowing the specific route. More powerfully, you can know that something is the wrong direction to go in (not strategic) even if you don’t know the specifics of the route you should take (the strategy).

Here are two example of how strategic and non-strategic things affect our daily lives.

Example 1: Non-strategic work leads to expense and pain

This is an example where doing something that is not strategic would have meant making our business more expensive and difficult to run.

In June 2008 we launched a new-look Comment is free (Cif) on our Java platform with Pluck powering the community features. Previously is was running on Movable Type. Now let’s work backwards a bit… the site was launched in June 2008, so anyone can guess that work might have begun in, say, January. We’ll have needed to sort out some legal paperwork with Pluck, which you can imagine might have taken three months to negotiate, and before that a market survey and possible prototyping to assess the alternatives to Movable Type (let’s say two months’ work).

So in August 2007 we would have been at point where there was general acceptance that Movable Type was not the future platform for Cif, even though we didn’t have any certainty about its specific replacement. Movable Type was not strategic, even though we didn’t have a specific strategy, or at least not a very detailed one.

And at this point any significant feature development on our (now non-strategic) Movable Type would have had three kinds of cost implications:

  1. Its lifetime value would have been limited;
  2. It would have created more work for the people migrating to the new platform, because they would have had to migrate more features;
  3. It would have added more support costs to the Movable Type platform and hence the company.

This last one is probably most relevant to technical people, journalists and the production staff who work on Cif. Because “support costs” means not just “money” but “time and pain”. With the focus moving on to the next platform there will have been less and less expertise available to work on the limited-lifespan technology: there will have been fewer people available, and knowledge would be less fresh. Any technical work would be more time-consuming to put in place. It would also be less reliable because reliable work (rather than quick hacks) generally takes more time and more expertise; and yet this is a platform with less expertise and a known limited lifespan, so greater reliability is especially difficult.

Example 2: Strategic work gives you stuff for free

Meanwhile, what of all that so-called strategic work on the new platform? Why was that “strategic work” and not just “work”?

Because so much investment (translation: time and effort) had gone and would continue to go into the new platform there are suddenly lots of mutual benefits. Cif got lots of things for free, because they had been developed before for previous launches on the same platform. Some of the front-end features I can spot which came free include:

Similarly other sections have gained from Cif’s launch. When we later launched the Life & Style section I was particularly pleased to see the editor had written her introductory article on the new platform and opened up comments underneath it. Commenting was previously available only on our Movable Type blogs, but by building Cif on our strategic platform the Life & Style editor got more features than she might otherwise have had.

Summary

Sometimes management jargon really is jargon. But sometimes it translates into a tangible reality. Strategic things are about creating a better working life for us in the future, whether “better” means less costly, less frustrating, quicker, or more flexible. That should make it meaningful for all of us.

Buzzmachine goes clunk: When media companies do tech

· 2 Comments ·

Jeff Jarvis has a typically provocative post, saying that newspapers should outsource their technology. Lloyd has already responded saying that there’s more to journalism than news-gathering, and previous technological mistakes should not close the door on future successes. Here’s my response.

In this article:

The original position

Jeff’s position is part of his book-to-be on thinking like Google, which we assume is going to be called “What would Google do?” He’s taking many different aspects of the (traditional) commercial world and asking how organisations should reshape themselves in the light of Google’s success. A central theme is aggregation, and that a much greater, cheaper, more valuable whole can be found by uniting and organising many disparate smaller parts. Or to, reword James Carvill’s phrase for Bill Clinton, “it’s the network, stupid”.

In his latest post the proposal is that newspapers should push their technology into the cloud and focus on news. Actually, this is a bit odd, because I don’t know of a newspaper which doesn’t consider itself part of a media organisation, and Jeff would certainly say they should do so, so I’m going to refer to media organisations instead of newspapers. I think this is fair. Jeff kicks off with a discussion from Edward Roussel of the Telegraph, which one might mistake as a newspaper, but is more correctly seen as a multimedia operation within the Telegraph Media Group.

Much of Jeff’s argument, however, comes from Bob Wyman of Google, who says

Your IT infrastructure is a COST of doing business. It is not a thing of value.

Today’s newspapers invest in their web sites out of vanity and from an inability to get their heads out of the geographically defined markets of the past.

And when prompted by Jeff he says, yes, it would make a lot of sense for publishers to hand over the technology to Google. Jeff then sums up his own position like this:

So take the advice, papers: Get out of the manufacturing and distribution and technology businesses as soon as possible. Turn off the press. Outsource the computers. Outsource the copyediting to India or to the readers. Collaborate with the reporting public. And then ask what you really are.

So that’s the position. And here’s why I disagree…

Argument 1: You’re asking the wrong people

This is the flippant response I’ve already posted on Jeff’s blog. Since the question posed is “What would Google do?” Jeff naturally dropped a line to someone at Google, and guess what they said? They said “Let Google deal with it.” Amazing! What do think the response would have been if he’d called up a Microsoft executive? “Let Microsoft deal with it” perhaps?

So let’s take Bob Wyman’s opinion for what it is: an intelligent and thoughtful response from someone who has already made his choice about which company he wants to work for. Then we can take that opinion, and use it alongside many others (perhaps including Microsoft’s) and find a conclusion that we think appropriate.

Maybe we should even rephrase the question. It’s not a foregone conclusion that we all want to be like Google. Maybe the question should be “What should we do?”

Argument 2: How much do you want people to change?

Deploying standard technology in an organisation is often perfectly good, and pushing technology into the cloud is often useful. For example, Google Apps as an commercial office suite is going to be perfectly usable and very cost-effective for many organisations.

But sometimes that technology isn’t going to be quite right for your staff. Maybe they regularly need to read complex Word documents from clients which don’t import very well into Google Docs; maybe you need to change a few processes in your business to better fit the technology; maybe your staff need to stop whinging and be reminded who’s boss, dammit. What approach are you going to take? How much are you going to ask your people to change, and how much should the company (and its technology) change around the people?

This is part of the reason we have our own content management system at Guardian.co.uk. We could have bought one off the shelf — a lot of similar companies did: The Times, The Jewish Chronicle just this week, The Telegraph with its blogging platform, and so on. These organisations seem to have decided that their staff would learn to work in a new way, with new workflows and new concepts, all driven by what the technology offered. (I say “seem to” because I have no insider knowledge.) This is fine, and change is good for people, if not always comfortable. By contrast our technology development is driven a bit more from people than to people, and hence we have a content management system designed very much around the needs of our journalists, sales staff and others.

All that probably says something about the various organisations’ cultures. If what’s in the cloud or on the shelf suits the organisation and its people then that’s great; if not, then it’s not necessarily right to force your staff to change for the sake of technology.

Interlude: How to build a media company in the cloud

Before the final push here’s a scene-setting interlude…

Let’s consider Bob Wyman’s list of Google’s offerings, and just for fun let’s run with the idea for a bit:

Heck, an online paper isn’t much more than a complicated Blogger.com. […] Google has search engines, alert systems, video serving, annotations, database services, AppEngine, more scalability than you can imagine, etc…. […] But, if Google doesn’t do this or, because of political issues can’t do it, then Yahoo! or Daylife or even the AP should do it instead.

So you’re just starting out in your media empire, you set yourself up on Blogger.com and start publishing. Then a while down the line you find you want to distinguish yourself and put a nice logo on your site. You find a graphic designer with Adobe Illustrator (that’s not really a techie skill) and they design a nice logo, and put it into your Blogger.com template because they know a bit of HTML and CSS (those are techie skills, but, hey, your 10 year old nephew can do that even if you can’t, so it’s not very techie).

Then you get an e-mail from an unhappy reader saying your site doesn’t render very well on Safari 3, and that’s a bit embarrassing. But now your graphic designer can’t help you, because they’re a graphic designer and they’ve reached the limit of their CSS skills. So you hire a professional web developer, and now that actually is techie, but at least they’re not a programmer…

…which is a shame really, because you’re suddenly finding you want to allow your readers to slice and dice some scandalous schools data, and you could make use of Google AppEngine if only you had someone with Python skills.

Some time later, in your now-bigger media group, you’re finding that for some months many of your services haven’t been performing terribly well and you’re losing traffic and reputation. After some pain, due to lack of in-house skills, you become aware that the problem is in the database. So at great expense you hire in a database performance expert who tells you that different databases need to all be tuned differently because they have different performance profiles, and Google’s own database services don’t offer the tuning you need for your particular setup. In fact, says the expert, the people who developed your current system had to write it in a really weird way just to get the performance they did, because they were constrained by the particular services that Google did and didn’t offer, so it’s now absolutely at the limit of its capability.

Never mind. You can always move it to Yahoo! or Daylife “or even the AP”. Obviously you’ll need to draft in a whole bunch of people. I’m afraid there will be some techies among them, and obviously none of the code in Google ports to any of the new services, so they’ll have to be rewritten; you’d have known that before you started all this, if only you had some technical staff of your own, but of course you didn’t because you’ve been concentrating on your core business. You’ll need some programmers, a project manager, and I suggest an architect, some quality assurance people and a couple of systems people, including a performance tester.

You might be wondering at this stage whether you can identify a suitably skilled group of people, and of course you would be able to do that if you’d built up a technical competence in-house.

You might also be wondering if moving to Yahoo! or Daylife “or even the AP” will leave you vulnerable once again to the limitations of their services. And well you might. And you might be wondering if your site will still end up looking like a hundred other blogs-with-knobs-on, or if you’ll ever achieve something like the New York Times, or El País, say. There is another option: to bring your technology much more under your control. It might not be right decision for some. But for you, it might be spot on.

Argument 3: Technical investment is inevitable

I hope that interlude shows that developing in-house technical capability is essential to any media organisation that is even fractionally ambitious. That capability will always include people, and sometimes it will include hardware or software. To make an up-front decision that your technical capability will never include hardware or software is pointlessly limiting.

You may argue that in the interlude above the media owner took a wrong turn. Perhaps they shouldn’t have introduced that schools data mash-up. Perhaps they were wrong to use Google’s database services in the way they did. Maybe they should have stopped before putting the company logo on their web page. But to argue any of those things is simply to argue that a media company shouldn’t be ambitious. It’s perfectly reasonable to argue that a media company can be too ambitious, but to set the bar at the same level for all such organisations regardless of their size, their history, or their people, must be a mistake.

On a more positive note…

Finally, let’s dispel the tone of opposition, because there’s one thing about Jeff’s argument I’m very happy about.

One of the central planks of Google’s success, and Jeff’s thesis, is harnessing the power of the network for mutual benefit. And Jeff doesn’t just talk the talk: he walks the walk, and he’s using the power of commenters to build the arguments in his book. (In one post he’s even quite explicitly soliciting input.) And that’s really great, because it’s all for mutual benefit. It’s advertisers who make Google’s services worthwhile, and Google shares that revenues with them, for mutual benefit. And because I know Jeff walks the walk I know he will share the success of his book with all those in the blogosphere who have built, tested and shaped it.

Jeff, I’m delighted to have contributed. I’ll be sending you my bank details momentarily.

Big lessons from a little project

· 4 Comments ·

I’ve just finished a fortnight’s holiday which I (foolishly) spent mostly in front of a PC developing a never-ending little application. But unexpectedly, despite the trivial nature of my project, I rediscovered a number of important lessons more usually associated with serious application development.

The software I’m writing is a just a little Firefox plugin. I’ve been fiddling with it for so long we’re almost onto the second major release of Firefox since I started, and yet it’s probably not much more than a couple of hundred lines long. You can see it really is a minor enterprise. Despite this it’s been quite a surprise — quite a shock, even — to be reminded of some industrial-strength truths in a small and personal environment.

And they are…

1. Damn, writing software is difficult

What we non-developers know about software we know either by observing or by talking to those who do it. But anyone who wants to be trusted won’t complain about their job or bore you with details they know you don’t want to hear, and so you’ll never hear about everything there is to know about software development, even when you ask.

So one thing I discovered — again — is that writing software is really difficult. Sometimes I was flying, but more often I was crawling: piecing together information from different sources, trying to understand what was possible, learning all kinds of technologies (Javascript, XUL, XPCOM), trying use them well, but more often trying to get them to work at all.

It’s made me respect all the more the people I work with every day who make it look so easy. Every line you write is pure logic and needs to work 100%. This is not like writing a letter or a sales proposal, where a sentence that’s only 95% perfect is more than adequate. This is like writing a legal contract from scratch for a particularly unpredictable world. Every line needs to be watertight.

Aspects of commercial software development2. Simple design really is difficult

Simple design is another way of saying that the software should be easy to work with and modify. You’d think this would be hard to get wrong, particularly with something which is being built as new: surely you just add small, simple pieces one at a time, with each one adding the next feature on the list. What could go wrong?

I found out first hand that simple design is difficult to achieve, even in very simple scenarios. For example, my application has a class which handles the user’s preferences. It needs to be initialised for when the application starts, and it needs to take and save the preferences when requested. All this takes place within a single object. So I wrote an initialisation function which set up the initial preferences according to how it’s been configured, I tested it, and all was well. Then I added some functions to take updated values and save them, tested those, and all continued to be well. Total: about 20 lines of code over four functions.

Then I ran the whole in an integrated environment and weird errors started occurring. It took me about two hours to figure out what was going on: it was a strange combination of unexpected start-up values, Firefox calling the initialisation function at unexpected times, and some confusing logic of my own which was supposed to protect double-initialisation. In the end I decided the best solution was to throw out the whole idea of initialisation. Now the object just takes and save preferences, and you can do that when the application starts if you want. Total: 5 lines.

The point of this is that an apparently simple and obvious design was actually too complicated to sustain. I’m very pleased that the solution was to delete lines and simplify. But I could only do this so easily because I had complete control of the code. In other circumstances there might have been other systems which were relying on that initialisation code (however flawed it may have been), and I might have had to add to the existing complications to solve my problem; or I might not have had the time to take a fresh look at the code and simply built around the flaws out of a sense of fear of touching the wrong thing.

This is a tiny example from a tiny piece of work, but it showed me how easy it is to go wrong with a design, and how easy it is to produce software that is complicated, hard to understand, and time-consuming to fix and evolve.

3. Learning a language is more about culture than syntax

I sometimes get CVs from people who claim to know about 10 programming languages, and I’m always doubtful. Just because you’ve written an application in a language it does not follow you’ve done it well. Knowing the syntax is only the first step. You also need to have good knowledge of any libraries, and finally you need to know how to work with the grain of a language. This means you’ll use different idioms, and structure your solutions in different ways.

In my own case I’ve been writing Javascript, but it stinks because I’ve tried to use it like Java. I’ve been stuck in my old Java ways, like creating classes and carving out a deeply-nested namespace. It’s Javascript, Jim, but not as we know it. It works, it makes sense, but it looks clunky and… well, it just feels wrong, dammit.

Javascript is a prototyping language. I can even tell you what that means, but only with my head, not with my heart. I use the prototyping as a hoop to jump through to get it to do the Java-y things that I know I shouldn’t be doing in the first place. Being a prototyping language doesn’t mean Javascript is a second-class language, or a dumbed-down Java. It means it’s a different kind of language to Java and should be treated as a first class language with its own ways of getting things done.

It’s a cultural thing, and you can’t claim to really know the language if you don’t operate comfortably in its culture. I don’t really know Javascript.

4. How did I ever live without automated tests?

Possibly by not spending my holidays sat in front of a PC. But aside from that, I continue to wonder at the marvel that is automated testing, and unit testing in particular. To be able to implement a change and not have to trouble your brain about the consequences is very liberating, allowing you to move ahead with confidence. It does take some work to set up the environment, but the results are worth it.

That said…

5. Your automated tests won’t cover everything

In one of my functions I unexpectedly found a truck-sized hole which had gone undiscovered despite seemingly comprehensive automated test coverage. (A loop which had a “break” when it should have had a “continue”, meaning great swathes of actions got skipped in most circumstances.) I only discovered this through integration testing (which is the fancy name I’m using for what was really “trying it out”), and found that a quirk in my unit testing setup had caused the mistake to be missed. Once I had found the cause I adjusted the main code to be more predictable and put in an automated test to trap the error, but it was only discovered through real hands-on testing.

6. A strong IDE sustains motivation as much as anything

Although I was using Eclipse for development, when working with Javascript it really doesn’t offer the comprehensive support you get with Java. Because Javascript is a dynamic language, and no doubt also because of the state of JSEclipse, there’s very weak support for code completion, refactoring and so on.

The consequence of this was a loss of code-writing speed, but much more than that I was suddenly able to see how easily a weak IDE allows bad software to be produced. There were many occasions when I knew that I should tidy up or refactor something, but was then suddenly hit with a premonition of the tedious steps I’d have to go through: working out which files to pick on, the manual search-and-replace, checking the context before I made a change. I had to force myself to get on with the tidying up despite knowing how painful it would be, focusing on the long-term results, and safe in the knowledge that for this little personal project I didn’t have a deadline.

It became clear to me that so many people must come under a constant barrage of pressure, with only their current strength of character to defend against the pressures of deadlines and short-term wins. It’s inevitable that too often they will give in to those pressures, leaving cumbersome code building up, and ultimately gumming up the works of the system. A strong IDE removes the barriers to those virtuous tasks of improving design and allows you not only to do your job, but to do it well.

7. Estimation is difficult

Which is just an excuse for the shameful truth: all my estimations were out by a factor of four. This is embarrassing because it’s not as if I’ve never written software before.

In retrospect the mistake I was making was to look at the component parts of a task, guess how long they’d each take, and add them up. What I should have done was try to take the forthcoming work, identify similar previous tasks, and from past experience see how long they actually took.

At the end of the day experience is good, but it’s how you use it that really counts.

8. You can’t know all your requirements up front

This is familiar to anyone who’s bought into Agile, but it hits home hard when it’s you who’s the user setting the requirements.

I’ve written countless requirements specifications in the past (in 60-page documents, on task cards, wherever) — I thought I really ought to be clear-thinking enough to know my own requirements up front. Wrong again. As the UI came together, as one idea sparked off another, and as I had chances to step away from the code to think about things from a distance, I started to see that my feature set was really rather disjoint — almost random. These were moments of clarity that on the one hand caused me to add requirements, but in doing so I was recasting the software with a new perspective. I was starting to see what the software should be doing, which was not quite the same as what I’d started out on.

Release date

Fortunately I’ve not sent out a press release announcing a release date, held a press conference, or hired the London Eye for a glitzy media event. I’m just writing software for fun, and at this rate it’s probably not going to ever see the light of day. But even then, it’s been startling to find that the germs of some of the Big Ideas of software development are still present in the smallest of projects.

QCon London 2008: A Michelin-starred deli

· 3 Comments ·

There were very few moments for me during QCon London 2008 of earth-shaking enlightenment — if any. But every hour of the three days of the conference there were insights and guidance that could be tucked away, and reused later to save hours, days or weeks of time elsewhere. Snake-oil salesmen where thin on the ground, and instead there were dozens of people saying one or both of:

  • This is what we did; and
  • This is what you can do.

No magic, no silver bullets, but plenty of solid advice and experience.

A good example of both of these was Randy Shoup of eBay. He had nothing to sell (other than the good name of eBay, perhaps) and his presentation was very clearly constructed to show their principles of scalability, and some concrete examples of how these work in practice. You probably wouldn’t use their periodic batch processing method to generate recommendations — if only because it’s odds on you don’t have a recommendation system — but you could take the overarching principle of “async everywhere” and apply that to the next scalable application that you need to work on.

Even the very specific presentations contained valuable points that could be generalised and reused. For example, Matt Youill and Asher Glynn of Betfair talked through how they scaled the transaction processing on their servers by a hundred-fold. Guardian.co.uk doesn’t need that kind of throughput, so the details were primarily of intellectual benefit. But a key practical lesson was how they approached the problem: by presenting it to industry players as a challenge carrying great kudos to the winning company.

All of this was summed up very nicely by the team from BBC News: John O’Donovan, Kevin Hinde and Ross Heritage. They were asked how they managed performance testing for the iPlayer. John spent a few moments describing some of the techniques they employed, but got to the point when he realised the audience really wanted some eye-opening enlightenment which he didn’t have. At this moment Kevin stepped forward and said straight out “There’s no secret sauce”. Indeed not: they just work hard and stick to strong principles.

QCon offered little in the way of secret sauces, but it did contain dozens and dozens of great ingredients you could take away and use to concoct your own wonderful dishes.

And with that analogy pushed to breaking point, I think we should leave it there.

Guardian.co.uk: Steps to a smooth launch

· 1 Comment ·

At the weekend the Guardian website went through one of the most significant transformations in its history: we moved our news, politics and Observer content into the new design and new content management system, and we simultaneously launched a lot of new functionality, both internal and external.

There’s an introduction and discussion on the more public-facing aspects of this, kicked off by Emily Bell. For my part, I want to talk briefly about one of the most remarkable behind-the-scenes aspects of it: how we got the weekend launch to go so incredibly smoothly.

The secret is that the weekend’s work was only the final step after a great many before it, all of which were safely out of the way before the weekend…

Guardian.co.uk global navigation bar1. Software release

The actual software was released some weeks ago, in early January. This means that by the time of the launch it had been in use for some time, almost all the lines of code having been executed several hundred (and in some cases thousand) times already, in the production environment.

Even that January release was only an enhancement of previous releases which have been going out fairly quietly over the previous few months. The latest one included new internal tools, and updates to tools, to support some of the new features that are visible today.

Guardian and Observer archive navigation2. Building the pages

Meanwhile editors, subeditors and production staff have had to learn to use those tools. They’ve been using them to migrate a lot of the older content into the new-format pages. You might think that could be done by machine, but that’s not the case. Since we’re also changing our information architecture — adding a lot of semantic structure where previously there was only presentational information — it takes an informed human being to know how to convert an older page into its newer format. A real person is needed to understand what a piece of content is really about, and what it isn’t about but merely mentions in passing. We also need people to look at the new keyword pages (for example, the one on immigration and asylum, or the page on Kosovo), which are populated mostly automatically, and for them to highlight the most important content: the most important content won’t necessarily be the newest.

This work had been going on for many weeks before the weekend launch. The January software release brought in some tools refinements to help them tidy up final loose ends (and no doubt some more tidying will happen over the next couple of weeks).

You can see from this that it’s about much more than mere software. The software enables people to do the editorial work, and the editorial work has been going on for some considerable time. Everything they’ve done has also been tested and previewed, which allows them to see what our external users would see were it to go live. Again, this exercises the software fully, but only within the internal network, before it’s exposed to the outside world.

3. Rehearsals

The work for the weekend launch is mainly running a lot of database scripts to make various new URLs live and decommission old ones. The reason this is such a huge launch is that there’s over ten years’ worth of news content to expose, as well as new functionality and designs.

We couldn’t trust the scripts to work first time (of course), so we spent a lot of time copying the production content into a safe environment, and rehearsing the process there, with real data. We needed to be sure not just it would work, but also how long it would take (considering any hardware differences), and change the scripts and process accordingly.

Guardian.co.uk favicon4. Launch

Finally, after all the rehearsals, the real deal. The work to run the database scripts and raise the curtain on various features ran over Saturday and Sunday, but it was calm enough and organised enough that the team needed to work only slightly extended working days.

So the big launch was a culmination of a huge amount of effort, and the weekend work was after an awful lot of practice. There were a couple of sticky moments, but nothing the team couldn’t recover from in a few minutes. As one of the developers remarked towards the end of Sunday: “The fact that today’s been really tedious is a good thing.”

What we can see now includes

What’s next…

We’ll be ironing out a few things over the next few days, but everything’s gone to plan for now. And then, as Emily says, there’s still sport, arts, life and style, and education to do.

The unstoppable urgency of web development

·

While I’m usually proud of the work I’m involved with, I’m rarely happy for long. There are always ways to improve, and I’m usually dissatisfied by one unmet ideal or another. Almost since I started in this field I’ve been vexed by how much of web development is “urgent” rather than “important”. This is not merely of theoretical interest; it vexes me because dealing with important issues gets you to something of lasting and strategic value, while dealing with urgent issues gets you to the next day. It’s that unmet ideal of achieving more long-lasting value that causes me to think about this.

I want to give some context around the claim that there’s more urgency in web development, and then I want to offer some ideas about why this might be. If you want the brief version, I think what makes the web generate such urgency is that…

  1. Updates are cheaper;
  2. Problems hurt less;
  3. Competition is more visible;
  4. Boundaries are blurred; and
  5. It’s easier to have more stakeholders.

But first, some context.

Picture will go here when I have timeSome context

Although I do think web development has a surfeit of urgency, it’s fair to say that “web development” is not a clearly defined area. So much software today has some link or other to the web, it’s usually unfair to say definitively that something is or is not web development. So it’s a matter of degrees, and looking back on the software development I’ve been involved with I’d say the more web-oriented something has been the more I’ve had to deal with urgent issues, and the less web-oriented something has been the more I’ve had to deal with important issues. The most urgency comes when working on websites, and the least when working on applications which may or may not use the web for a bit of minor communication.

This issue of urgency also seems to be the perception and expectation of non-technical people. This is how we get the phrase “Internet time” and why for years I’ve heard people saying “but this is the Internet” as a justification for tight deadlines. This is something that continues today, even after the unsustainable boom and bust of the dot.com years.

When urgency is too pressing we risk doing a bad job, and releasing something which isn’t up to scratch. Of course, when urgency isn’t pressing enough we lose ground to our competitors. And of those two scenarios it’s the first one, doing a bad job, which is easiest to spot: a bug bites someone on the nose and you get to hear about it pretty fast. The second scenario, losing to the competiton, happens slowly over months or years, and by the time you slip into obscurity so many events have occurred it’s hard to pinpoint exactly where you went wrong.

Urgency is no bad thing, then, but I continue to be bothered by its dominance in web development. Too much urgency leads to a loss of quality and a reduction of strategic actions. Why does this happen in web development more than elsewhere? Here are some thoughts…

1. Updates are cheaper on the web

Rolling out a new version of a website onto a server shouldn’t be regarded as a trivial affair, but it’s a lot easier than other means of releasing software. So it’s easier to release a slightly rushed product if we know the only copies anywhere in the world live on servers which are entirely under our control.

Slightly more costly than updating our own servers is the job of the IT department which has to update an application on everyone’s desktop. Here, not only do you have to be sure the desktop machines are all switched on and ready to receive the software, but an error for one person could mean they cannot work at all the next day. More costly than that is the job of burning, boxing, and shipping a DVD to hundreds of retail stores.

For both those cases, and for many other ways of releasing software, the cost of repetition and the cost shipping something substandard is very high. But in many web applications, where the cost of distribution and the cost of error is relatively low, the consequences of problems that come from an urgent release are much less costly.

2. Problems hurt less on the web

As users, our relationship with websites and web applications tends to be less engaged than with most other applications. So although we all know undue urgency leads to more faults being released, it’s perhaps more forgivable on the web.

If I’m using a website and I encounter an error, I know the problem — the website — is theirs. I can come back later or try an alternative (there are always plenty of alternatives on the web). Either way, I’m probably not going to get very upset.

By contrast, if I encounter a problem with software running on my machine then any error is going to hurt me much more: that’s a problem with my software… it’s causing my machine to go wrong… I went to the trouble of installing that software and now it’s giving me problems… this is not good. Even if the software was installed by my IT department it’s still my software, and it’s core part of my daily work — finding a workaround is going to be difficult.

3. Competition is more visible on the web

One of the drivers of urgency is beating the competition, and if you and your competitors all work on the web, then they are likely to make (cheap) incremental releases, and you will notice almost every one.

If competitor A releases a new feature in week 1 then you’ll notice and ask how you can do that yourselves. If competitor B releases another new feature in week 2 then you’ll notice it and ask how you can beat that, too. By the time competitor C releases a third thing in week 3 you’ve got the pressure of the three things being delivered by your competition and nothing to show yourself.

Things are different as you move offline. Consider software companies in a competitve marketplace that see software ship features in batches. When your competition ships their new release with new features you’ll be able to assess all those features as a whole. Of course there will still be intense pressure to deliver, but your view will be more rounded and decisions of importance will play a greater role than decisions of urgency.

Consider also the internal team which delivers software which is internally-facing. Your life in this team will be no less demanding than in any other, but one thing you’ll be spared is much pressure from competitive teams. If you’re delivering the next version of the payroll system then there will be pressure of deadlines, there will be pressure of budget, but you’re very unlikely to get an unexpected new requirement due to the payroll software team at your competitor suddenly releasing their new e-mail alert module.

None of this is to say that competition is more fierce on the web. That may or may not be the case, but it’s not what I’m interested in here. The point is that the competition seems more fierce and is much more visible. This encourages urgency.

4. Boundaries are blurred on the web

A wordprocessor helps you write documents, a media player plays audio and video, but a website… a website doesn’t have clear boundaries.

Although I wrote above of competition, exactly who is a competitor on the web is not always certain. I once worked with an e-commerce music retailer who suddenly decided they needed to provide a web mail service for their customers. I’ve worked with a law firm who wanted to turn their brochureware site into a recruitment platform. Yahoo! evolved from a directory into a portal, embracing almost everything they could while the world struggled to understand what a portal might not be. Meanwhile, stepping away from the web, I don’t know of a spreadsheet which carries a discussion forum or an HR system which recommends movies.

While all this clearly accounts for the quantity of demands that occur in web development, it also accounts for the urgency because that steady drip-drip-drip of features that comes from our competition also comes from those who aren’t direct competition but are just out there on the web. The drip-drip-drip becomes a trickle (and maybe even a flood) pretty quickly, and the pressure to deliver new things is even greater.

5. It’s easier to have more stakeholders on the web

Because the web means so many things to so many people — contributing to those blurred boundaries — many more people within an organisation will consider themselves stakeholders. Marketing people see it as a marketing tool, the recruitment team see it as a recruitment tool, the sales team see it as a sales tool, and so on. And of course each one of them is right: it can be all these things.

With so many diverse stakeholders, we can expect so many more pressing deadlines, and hence much more urgency. More stakeholders also means communication and prioritisation is more difficult. It’s difficult to get everyone into the same room, it’s difficult to bring everyone to the same point of understanding, it’s difficult to weigh completely different kinds of requirements against each other. It doesn’t matter that everyone is working for the same company and therefore, at a higher level, the same goal. That goal is usually at too high a level to make a difference to everyone’s day-to-day needs.

The least pleasant example of this I remember is a colleague at a previous company who was the project manager for one of our clients. Although he was their single point of contact they managed to have several single points of contact, none of whom were very good at speaking to the others. He found he had responsibilities to the marketing manager, the distribution manager and the IT director. They all had pressing demands, and he had no contact with anyone who could bring them all into line. Indeed, that seemed to be by design, as we suspected the client used this as a way of squeezing us as a supplier. The relationship did not end happily. Although that’s a very extreme example it does highlight the kind of difficulties of having many stakeholders.

A final word

Cheaper updates, less damaging problems, more visible competition, blurred boundaries and more stakeholders. This tends to be the world of web development, and therefore a world in which urgency plays a more dominant role.

Yet we can think of web development projects in which importance dominates urgency. One which springs to mind is an online banking site: this is web development if ever anything is, but one in which doing things quickly will always be trumped by doing things properly (whatever “properly” may mean for that bank). But this is also an environment where many of the criteria I’ve listed don’t hold. Most obviously updates will be fairly costly, any problems will be very damaging, and the boundary of what the site does is very well defined.

So the criteria I’ve set out above are more to explain than define. Web development tends to take place in an environment that’s different from many other software development environments, and I think it’s those features of that environment that I’ve listed that explain why urgency predominates.

I said at the beginning this vexes me because urgency causes visible problems, and eliminating urgency would seem to help eliminate those problems. But I also said I’m usually dissatisfied with one unmet ideal or another. I’ve also worked in environments where there is a distinct lack of urgency. That, of course, leads to waste and stagnation. And that’s really, really, vexing…