As part of a project some new requirements came in. I had been concerned that part of the system under development was a little complex, but not overly concerned, as it worked and had comprehensive automated tests.

But the new requirements changed that. They concerned a subsystem which took three inputs and processed them. The problem that the subsystem was trying to solve was roughly stated as follows:

We have an input stream which needs to be combined with two other input streams. But if some of the combining can’t take place because the input streams have gaps then we have to fill in the gaps from other sources. And then we can output the result.

If that seems a bit vague, it’s because it is: I hadn’t paid much attention to expressing the logic clearly.

The new requirement, meanwhile, was to add a fourth input. Since we were using a visual tool I can present the “as is” software design along with the “to be” that was the likely outcome of the requested changes:

The threatened “to be” state would have been excessively complex. Two things concerned me greatly. First was that I couldn’t keep all the “to be” logic in my head at one time, so didn’t have confidence that it was right. And second was that — as a guiding rule — software should be exactly as complex as the problem it attempts to address, and no more so… and that was not reflected here. My gut feeling was that the problem was simple, and adding one more input should have added linear complexity, but it felt like the complexity was growing exponentially. This was compounded by the fact that it was quite apparent there might soon be a fifth, sixth, or even seventh input.

The solution was to take a step back, express the logic of the problem clearly, and redesign the subsystem to reflect that. The problem, once clarified, became this:

We have a number of input streams which are merged. For each missing element of the merged stream we take action to fill it in. And then we output the result.

The refactored “as is” and the eventual “to be” logic became much simpler:

I hope you agree with me that the new flow is much clearer, and additional inputs add only linear complexity.

In terms of software development the sequence of steps was: (1) Make sure all the tests pass, (2) refactor the subsystem to reflect the new logic, (3) make sure all the tests still pass, (4) add the fourth input with appropriate tests.

For me this experience demonstrates a few things:

1. If software internals aren’t continually kept clean and of good quality then complexity increases excessively and progress slows accordingly. Or to put it another way, technical debt is very well named: the longer you neglect the debt the more interest you pay.
2. “Software should be exactly as complex as the problem it’s addressing” is a very good driver for reducing complexity.
3. Automated tests save the day again! The refactoring above would have been quite daunting and unreliable without them. But instead the worst thing about the operation was that it was a little time-consuming. It was not risky, it was not stressful, and it was not really that difficult.
4. Visual software development tools sure do expose messy software design for what it is.

There are a few pertinent lessons:

1. Roll out slowly
2. Let systems co-exist
3. Always have a rollback plan

1. Roll out slowly

I’ve written before about the almost-inevitable pain of big bang launches, following Telegraph.co.uk’s update in 2007. Those lessons are true still. The only way to avoid that pain while still having a big bang launch is with a vast amount of additional resources (cost and time), which are rarely worth the value of what you’re gaining.

An approach with more likely success is to roll out piecemeal, replacing one user feature, or one system component, at a time. This is not sexy for engineers, who typically want greenfield projects and new technology, but nor is it sexy working 18 hour days to fix a torrent of bugs under a hail of angry user feedback.

This approach means breaking up the system into discrete components and replacing those. If the system doesn’t have such discrete components then the first job is to create them, going through the code and making logical fences. No this isn’t sexy, but let’s repeat: neither is working 18 hour days.

2. Let systems co-exist

It’s a computer science joke that every known software problem can be solved by adding an extra layer of indirection. However, sometimes that’s also a practical solution. This allows you to build an old and a new system side by side, and funnel specific users/activity/processes/etc to either one. As you build up the capabilities of the new system you can funnel more and more activity to it.

Two architecture cast studies:

In the first (Architecture 1, above) we needed to replace a proprietary application server with one over which we had much more control. Critically, the new application server needed to be able to execute the same scripts, of which there were thousands.

Our first-release replacement app server had only the most basic capability, and so could only execute a tiny fraction of the scripts. So we put in a layer of indirection which filtered requests to either the old or the new server. As we enhanced the capability of the of new application server we filtered more and more requests to it, until we were finally able to switch off the old one.

In the second (Architecture 2) we needed to migrate our user data to a new, more flexible database (MongoDB), despite some of the application code being rather old and tied closely to the legacy database. The solution was to introduce an internal API, ensure all code used that API instead of direct database access, and then add the new database under that. The two systems co-existed for a long time as their capabilities were brought together and tested for consistency.

3. Always have a rollback plan

Once we have a step-by-step approach to moving from one system to another we have to recognise that some steps may not work flawlessly. In this case we have to be able to switch back, because fixing a problem in situ may take too long. This is something that can be built into the indirection layer’s config system, or into the release mechanism (allowing easy rereleaes of the previous version), or be switchable in the application’s own config  (see Flickr’s approach as one example).

It’s still difficult

None of this is easy, of course, but it’s the kind of thing were the effort is in the up-front planning and the initial (possibly time-consuming) prep to tidy up the old system, all for the sake of a smoother rollout. Launches are exciting, but they can be exciting in the intended way at the same time as going boringly to plan.

Meanwhile the folks at Delicious are slowly restoring many lost features, and I wish them well, because we’ll all win if they do.

Social decisions in software architecture

The other day in conversation about team structures the Guardian’s lead software architect, Mat Wall, mentioned that architecture is social. This is a good, and often disregarded, observation. In that context he meant that architectural decisions influence who works with who, what issues they need to sort out together, how they regard their work, and so on. Three examples…

The Guardian has an API that allows access to its content. That means that when developing an iPhone application the iOS team can work much more closely together, requiring less information about the guts of the back-end, and become more focused on the user experience.

Consider also the team embracing Scala. Our platform lead Graham Tackley has gone on record to say that one consequence of this is that it has “reinvigorated the team”. That’s not the only reason to take up a technology, but it is a significant consideration.

Third, I’ll relate the story of one of our recent projects. The team devised four candidate architectures and needed to choose between them. The first decision was to make the call in conjunction with the business owner. The second decision was for them to ask not “which architecture do you prefer?” but rather “what do you want your business to look like?” Because each architecture had a different impact on the end users inside and outside the company: certain things would be difficult, others would be easy, etc.

Social decisions in algorithms

And it’s not just about architecture and software design. It’s true also of the more abstract matter of algorithms, as seen by a couple of examples from the masters (or should that be “slaves”?) of the algorithm: Google.

The most recent example concerns Google+. Rocky Agrawal wrote:

I finally blocked Robert Scoble in Google+. I have absolutely nothing against Scoble. I quite admire him, actually. He’s a great asset to the startup scene and he works damn hard. I’ve met him a few times and I’m sure we’ll meet again. But he was just getting to be way too much.

My Google+ feed was dominated by him. I tried to take a half-step and just remove Scoble from my circles. But then he became Google’s perpetual #1 suggestion for a new friend.

Google have created an algorithm which provides recommendations, and of course highly-referenced people will be recommended more than others. But Robert Scoble’s star power is clearly so significant it’s ended up disrupting Rocky’s experience, and getting in the way of him using the service effectively. Clearly the algorithm needs to be tweaked for the optimum user experience, and how that is done is entirely at the discretion of Google engineers. They could cap the influence of exceptional stars like Scoble if they wanted. They have choices, and ultimately those choices are based on social, human instincts.

The second Google example comes from the spat earlier this year between the Google search team and Bing. Google noticed that Bing was copying some of its search results; they demonstrated this by deliberately generating irrelevant Google search results for particular nonsense queries, and then observing that the same results appeared in Bing for the same queries; Microsoft said, sure, “We use multiple signals and approaches”, including observing what search results people click on, even if those clicks happen to be by Google engineeers conducting a sting operation; Google cried foul; the world moved on.

What was happening was that the Bing toolbar was tracking search results that the user clicked on. If a user had the Bing toolbar installed, did a Google search, and clicked on a link, then the toolbar would send a message back to Bing HQ saying “Hey, someone thought this link was good for this query” and Bing would consider that next time it needed to respond to the same query.

I found the whole episode quite amusing and slightly baffling, most notably these rather indignant words on the Official Google Blog:

At Google we strongly believe in innovation and are proud of our search quality. We’ve invested thousands of person-years into developing our search algorithms because we want our users to get the right answer every time they search, and that’s not easy. We look forward to competing with genuinely new search algorithms out there—algorithms built on core innovation, and not on recycled search results from a competitor.

To be fair, I think that post was written before Google knew exactly how Bing was doing its job, but even so, they jumped to conclusions. My personal reading of the above is that “pure” algorithms using only digital, abstract data are good and acceptable, while algorithms that use human data and feedback from a wide variety of sources are cheating and bad.

Unfortunately that’s just wrong. Google may like to say they don’t editorialise and their search results are purely machine-generated. But what they choose to feed into their algorithm, and how they weight those things, is an entirely human affair influenced by very human traits: culture, prejudice, politics and more. That’s not a critique of Google or its staff; it’s a critique of human beings.

Jonathan Stray has made this point, too. He said “It’s impossible to build a computer system that helps people find or filter information without at some point making editorial judgements.” And he quoted Matt Cutts of Google: “In some sense when people come to Google, that’s exactly what they’re asking for — our editorial judgment. They’re expressed via algorithms.”

So however “pure” we think a technical decision might be, ultimately it’s not. Whether it’s software architecture, algorithms, or something else, it’s all influenced by and influences our social world.

1. The point of it
2. The no-tech solution (and credit)
3. The software proposal

The point of it

Here are example situations we want to improve in project proposals:

• “I’m very confident this will bring in $0.5m to $2.5m of new business”
• “I’m very confident this change will see no less than a 2% drop-off rate in conversions”
• “I’m very confident the addressable market is between 5m and 9m users globally”

We want to turn those into:

• “I’m 90% certain this will bring in $0.5m to $2.5m of new business — and you can be sure that when I say 90%, I’m right 90% of the time”
• “I’m 95% confident this change will see no less than a 2% drop-off rate in conversions — and you can be sure that when I say I’m 95% confident of something I really am right 95% of the time”
• “I’m 90% sure the addressable market is between 5m and 9m users globally — and when I say I’m 90% sure of something I really am right 90% of the time”

The purpose of the application is to help people really be right 90% of the time when they say they’re 90% certain — or any percentage they want.

Estimating well is important for accuracy. This is particularly true when we consider that targets like revenue and reach are actually factors of many other measures (market size, consumer price sensitivity, economic conditions) which are themselves estimated. With estimation piled on estimation it becomes even more important to move from a nebulous “very confident” to a concrete “90% confident” and so be sure of our bounds of certainty.

The no-tech solution (and credit)

Credit for switching me on to the problem, and for the no-tech solution that follows, is to Doug Hubbard and is part of the narrative of his book How to Measure Anything. What is his approach to getting good at estimating percentage certainty? Have a look at this question:

What is the height of the Eiffel Tower in feet?

I don’t suppose you know… exactly. I certainly don’t. But we can have a good go at estimating a range with 90% certainty. I am absolutely certain that it’s taller than me, so it’s definitely over 6 ft. And I’m virtually certain that a mile is a thousand-and-something feet, and it must surely be less than a mile high, but let’s say it’s less than two miles for safety.

So I’m 99.9% certain it’s between 6 ft and 2,000 ft tall. But that’s too conservative, because we’re aiming for 90% certainty, not 99.9% certainty. So I’ve got start using some other tricks. I’m 6ft 3in, and a house is about six mes tall… call it 40ft. How many houses tall is the Eiffel Tower? Between six and twenty, I reckon. So between 240 ft and 800 ft. That seems better — I can sign up to that range with 90% confidence.

Now let’s keep going. Answer the following questions, each with a range you’re 90% confident of:

1. What is the height of the Eiffel Tower in feet?
2. In what year was Anne Boleyn born?
3. What is the population of Tokyo?
4. How many square miles is Peru?
5. How many saints did John Paul II canonise?
6. How many books in the Mister Men series?
7. What is the weight of the Liberty Bell, in kilograms?
8. What is the circumference of the moon, in kilometres?
9. How many square centimetres is the Mona Lisa?
10. How many patents did IBM lodge in 1999?

The aim, of course, is to have 90% of your ranges right. This is what Doug Hubbard calls being “calibrated” — i.e. when we say we’re x% sure of something we are indeed correct x% of the time.

Unfortunately it’s not quite as easy as saying we must score 9/10 in the above questions — we’re expected to get 9 out of 10 on average, not necessarily 9 of these particular 10 right. So we refer to statistics, and the binomial distribution says if we want to be 95% sure that we are calibrated then we can expect to get 8, 9 or 10 out of 10. Similarly…

• 20 questions implies 17-19 should be right at the 90% level.
• 25 questions implies 21-24 should be right at the 90% level.
• 30 questions implies 25-29 should be right at the 90% level.

We can calibrate ourselves with true/false questions, too. This time we have add how certain we are of our choice:

The first Cannes Film Festival was held in 1961. True or false? And how sure are you of your answer: 50%? 60%? 70%? 80%? 90%? Or 100%

Saying we are 50% sure means we have no idea — equivalent to the flip of a coin. 100% means we’re absolutely certain. If we answer five questions with certainties of 60%, 60%, 50%, 90% and 80% then we expect on average we’ll get 0.6 + 0.6 + 0.5 + 0.9 + 0.8 = 3.4 out of 5. If we want to be 95% sure that we are indeed calibrated then (if my maths is right) we should expect to get 2, 3, or 4 out of 5. This is clearly too wide a margin to be informative so in reality we should again be answering 20 or 30 questions at a time.

By putting in a feedback loop we can improve. We answer 30 questions, find out if we’re over-cautious or under-cautious in our estimating, then consciously compensate while we try answering another 30 questions. And so on until we start hitting our targets.

There are also tricks to helping us judge our understanding of 90% confidence. Again, I’m taking my queue from Doug Hubbard. Take a look at a wheel with a one-tenth slice marked out and a spinnable needle in the centre. If we are 90% confident of something then we should be equally confident that we can risk £5 that a spin of the needle will end up in the larger zone (if it ends up in the smaller slice we lose our money). If we would rather spin the needle then we’re less than 90% confident of our assertion and should widen our range; if we’d prefer to stick with our assertion then we’re more than 90% confident and should narrow our range; it’s only when we think we’d be equally happy with one or the other that we’ve got to 90% confidence.

This seems to be worth trying with each question whenever in doubt.

I think being good at estimating like this would be a cool party trick. Especially if we go to the parties held by our local maths department. Now the only question is: where do we find an endless source of semi-obscure questions? This brings us to…

The software solution

The proposal, then, is for an application which automates this estimation training:

1. How many questions would you like to answer? 10, 20, 30…? And would you like questions about ranges or true/false questions?
2. Here are the questions, enter your answers. (If you want help with an answer you can pop up a wheel with a suitably-proportioned slice marked out — do you prefer your answer or prefer to spin the wheel? Now you can rethink your answer in the light of that.)
3. Feedback: You got this many right, which was/wasn’t what we expected for someone who is a good estimator. You’re over/under/correctly confident so you need to adjust accordingly/congratulate yourself. Now let’s go again…

I imagine this would look nice with now-de-rigueur CSS3, pastel colours and big shiny buttons with rounded corners. And a good source of questions is Wikipedia.

In particular the info box on the right hand side of a many articles is good source of stark facts — for example, the one for Chippewa County, Minnesota. There are some good range-type questions to be extracted from this: total area in square kilometres, population as of 2000, year of founding. There are also some good true/false questions (largest city, date founded) but only if we can find a reasonable-sounding false answers.

That last problem can be solved like this: First, at the bottom of the page find a list in which this subject appears. In this case Chippewa appears in a list of counties. Then find another item in the same list — say, Murray County — and go to that page and find its info box. Now we can compare the same aspects of Chippewa and Murray (largest city, date founded, timezone) with different values (largest city and date founded are different; but avoid timezone as it’s the same for both) and swap them randomly.

The list these two subjects were drawn from (Counties), the heading of that list (State of Minnesota) and the aspect selected from the info box (largest city, etc) provide a reasonable way to present the question, since English-language question generation looks a bit tricky. We end up with something like:

• Answer these questions with 90% confidence:
  • State of Minnesota > Counties > Chippewa County, Minnesota > Area – Total > ……. to …… km²
  • …
• Answer these questions as true or false, and state your confidence:
  • State of Minnesota > Counties > Murray County, Minnesota > Founded > February 20, 1862. True or false? 50%? 60%? 70%? 80%? 90%? 100%?
  • …

There are still some flaws with this. You have to go through quite a lot of random Wikipedia pages before you find one with a decent info box. And sometimes whether the info in the info box is true or false is too obvious:

• State of Minnesota > Counties > Chippewa County, Minnesota > Named for > Chippewa Indians, Chippewa River. True or false? 50%? 60%? 70%? 80%? 90%? 100%?
• State of Minnesota > Counties > Murray County, Minnesota > Website > www.co.chippewa.mn.us. True or false? 50%? 60%? 70%? 80%? 90%? 100%?

Maybe we should restrict the subjects to some known areas (books, US states, composers,…) and not look in the entire pool of Wikipedia articles. And maybe some kind of text matching might help avoid the embarrassingly obvious true/false questions.

Or maybe you can think of a better source of questions.

If so, there’s a very useful service just waiting to be created…

Footnote

Where to find answers to the above questions…

1. http://www.discoverfrance.net/France/Paris/Monuments-Paris/Eiffel.shtml
2. http://www.bbc.co.uk/history/historic_figures/boleyn_anne.shtml
3. http://www.metro.tokyo.jp/ENGLISH/PROFILE/overview03.htm
4. http://www.wolframalpha.com/input/?i=how+many+square+miles+is+peru%3F
5. http://en.wikipedia.org/wiki/Pope_John_Paul_II
6. http://www.themistermen.co.uk/mrmen.html
7. http://history.howstuffworks.com/revolutionary-war/liberty-bell.htm
8. http://solarsystem.nasa.gov/planets/profile.cfm?Display=Facts&Object=Moon
9. http://en.wikipedia.org/wiki/Mona_Lisa
10. http://www-03.ibm.com/press/us/en/pressrelease/1920.wss

Mat Wall on The evolving Guardian.co.uk architecture

Mat’s presentation was structured around rebuilding guardian.co.uk over many months and dealing with the scalability issues that were presented at various stages. Some of the many insights were presented in a deliberately provocative way, such as this one:

Developers try to complicate things; architects simplify them.

I’m not entirely sure about the first part, but it’s certainly true that over-engineering is a much more common trait than under-engineering. The second part, though, is very important. It’s something I’ve always known, but never quite in that pithy way. It’s a useful phrase to tuck away for future reference.

And if you think Mat wasn’t feeling too good towards developers, don’t worry, because he also put up a slide saying

Developers are better than architects.

The message behind this is that it’s the developers who are working on the code each day — they know what’s possible and what not, and what’s reasonable and what’s not. So it’s important to trust them when they say they can or can’t do something.

Dirk-Willem van Gulik on Forging ahead – Scaling the BBC into Web/2.0

Dirk-Willem talked about the programme to re-engineer the BBC websites to enable greater scalability and dynamic content. In an organisation the size and complexity of the BBC you need to spend more time thinking about people than the technology. One of his first slides underlined this by presenting the seven layers of the OSI model (physical up to application) and then said

But most people forget there are two layers on top of that: level 8, the organisation, and level 9, its goals and objectives.

From that point on he kept referring to something being “an L8 problem” or “a level 9 issue”. It was a powerful reminder that technology work is about much, much more than technology.

Another great insight was how much they have chosen to use the simplest of standard internet protocols to join various layers and services within their network — even when those layers and services are organisation- and application-specific. This ties back to Mat’s point about an architect’s job being one of simplification.

Phil Wills on Rebuilding guardian.co.uk with DDD

Phil talked about the role domain driven design has played for us. He also pointed out various lessons that there were learnt along the way, such as the importance of value objects, and the fact that “database tables are not the model — I don’t know how many times I can say this.”

But it was only a day or two after his presentation that I was struck by a remarkable thing: Phil referred so many times to specific members of the team, and talked about contributions they had made to the software design process, even though the people he was talking about were generally not software developers. He put their pictures up on the screen, he named them, he told stories about how they had shaped key ideas. This was an important reminder to me that being close to our users and stakeholders on a day-to-day basis is incredibly important to the health of our software.

Walking away

I didn’t get to see as much at QCon as I’d like to have done, although I dare say most people will say that, even if they went to as many sessions as was physically possible. But what I did see was fascinating and thought-provoking, even when it came from people I work with every day.

When you have a huge number of people for whom you are building software (1500 staff, 20 million unique users, and an entire wired economy influencing which way you should go next) then simply following instructions is insufficient, because your users’ demands will change and evolve over time — even if not during the current project then certainly before Version Two. So to minimise the cost of those changes you need to understand the way your users are thinking.

That’s where domain driven design (DDD) comes from. It’s about taking the concepts in your users’ heads and embedding them straight into the software you’re writing. And then when those concepts evolve and change the cost of changing your software is directly proportional to the mental shift that your users are making. When your users say “I want to make a small change” then usually it’s a small cost; if they propose a big change then they should understand when you walk them through all the implications.

For the R2 project we used DDD from the start, and it was key to many of our successes: when we discovered new opportunites which arose only from direct use of what we had implemented, then DDD allowed us to realise them.

Take, for example, the idea of “tone” — the principle that we should be able to categorise content by its “voice” or “style” (well, its tone). Its tone might be obituary, blog post, match report, and so on. The vague notion of this had been around since the start of the project, but we hadn’t settled on many details, let alone how to implement them. But after a few releases, and when the software started getting real use, it became apparent that applying a tone should be very like applying a keyword. Suddenly things fell into place. The functional requirements were clear, as were the implementation details. Tone has become a very powerful feature (here are all our obituraries, all our blog posts, all our match reports,…) yet it was a relatively straightforward piece of work because it was, in the end, a relatively intuitive idea.

Of course, domain driven design has a lot more to it than I’ve described here, and our use of it has been much deeper than feature implementation. If you’re lucky enough to be going to QCon San Francisco then today you can see our software architect Phil Wills presenting a lot more detail there.

This article is an attempt to explain what lightweight really means, both in terms of benefits and drawbacks. There are also a couple of comparative examples from my own experience.

Lightweight doesn’t mean simple

People often mistake “lightweight” to mean simple or quick. But this can’t be right, because everyone wants simple and quick, and if it really meant this no-one would use anything else. Every website would be rewritten with the lightweight Ruby on Rails and every application would be sitting on top of the lightweight SQLite database. Who wouldn’t? Who doesn’t want simple? Who doesn’t want quick?

Lightweight is often good, but it must have its tradeoffs, otherwise other technologies wouldn’t exist.

From the examples below I see lightweight as offering low cost in return for low demands, high cost for high demands. Heavyweight is disproportionately high cost for low demands, but low cost for high demands.

Lightweight carries low inherent management costs. But some situations require a high degree of management control whether you like it or not. That means that if a lightweight system needs to scale up you have to wrest management from it and maintain it externally. If you can do that then the lightweight system continues to work, but if the lightweight system will not relinquish management control, or if you don’t have the discipline to keep the management going, then it won’t be effective in the long run. By contrast heavyweight systems impose management and structure of their own. This is good if you’re going to need it, as it takes the pressure of discipline off you, but it’s not effective if you didn’t need that management structure in the first place.

To illustrate this, here are a couple of lightweight/heavyweight comparison case studies…

Language example: TCL and Java

TCL is a lightweight language. You get to write Hello World in one line, it doesn’t force much structure on you, and it’s pretty relaxed about how it’s written.

TCL is so good, in fact, that was the basis of the original Guardian Unlimited website. We built Ajax-style tools with it before Ajax was known as a concept, we generated our front page from it, we used it to integrate with our ad server.

But as our site grew the language didn’t scale with it. Clever shortcuts implemented by earlier developers confused newer developers because they obscured the purpose of the code. The lack of an imposed structure meant every foray into older code involved learning its idiosyncracies from scratch. Development slowed down as we worked around older code. And when we wanted to redesign the website we found that through years of lightweight flexibility we had allowed ourselves to be tied into knots: it would be more effective to start again than to work with what we had.

In fact, for the most part we’re now using Java…

In contrast to TCL, Java is pretty heavyweight. Not only does Hello World require three lines (excluding any lines with just braces), but its philosophy of structure and layering percolates through from the core language to most of its add-ons. For example, to parse an XML document you have to drill through two abstraction layers before you can find the parser.

One Java framework that maintains this ethos is Hibernate, used for database access. Its architecture is complicated, and as usual this is to offer flexibility without relinquishing manageability. Recently a forthcoming release of the Guardian Unlimited website was failing its pre-production performance tests. Our developers tracked down a major cause of the problem to an inefficient query within Hibernate. They extracted some of the query’s logic up into the application layer and simplified what remained, rebalancing the work between the application and the database. Problem solved, performance restored. What’s relevant to our story is that the developers did this entirely within the archicture of Hibernate, so they didn’t compromise the design of the application and therefore didn’t add complexity.

CMS example: WordPress and the GU CMS

Over on ZDNet Larry Dignan extolls the virtues of WordPress and says, effectively, “What have big content management systems ever done for us?”

WordPress is the lightweight CMS I’ve chosen for this blog, and I’m very happy with it. It’s easy to install, requires almost zero maintenance, and lets me focus on the writing. And yet I’m a strong advocate of the home-grown CMS we have for our journalists and subs on the Guardian Unlimited site. Is lightweight not good enough for our journalists? What has a big CMS ever done for us?

Well, I just looked at a current article on guardian.co.uk: “Ministers ordered to assess climate cost of all decisions”. It was created with our big CMS. What’s there that WordPress couldn’t deliver?

For a start it’s got a list of linked subjects down the side, which aren’t the same as WordPress’s tags because they’re tightly managed to ensure consistency and reliability. These subjects are also categorised, so Pollution and Climate Change are subjects under Environment, while Green politics is a subject under Politics. As I write this, I note also that the pages for Pollution and Climate Change are designed differently, with Climate Change being more pictorial and feature-led. Subject categorisations and subject-specific designs are beyond what WordPress’s tags do.

Okay, so apart from the linked subjects, the categorisations, and the subject-specific designs, what has a big CMS ever done for us? I suppose it’s worth mentioning the related advertising, which as write includes a large ad for environmentally-friendly washing liquid. There are other contextual commercial elements, too, such as the sponsored features, links to green products and books, and offers of reducing energy bills and offsetting carbon emissions. And there are related articles and related galleries. And details of the article history, listing when and where it was first published, on what page and in what newspaper section.

Okay, so apart from the linked subjects, the categorisations, the subject-specific designs, the related advertising, the contextual sponsored features, the links to relevant products and books, the complementary offers, the related articles, the related galleries, and the article history, what has a big CMS ever done for us?

Well, I suppose it is serving to over 17 million unique users a month…

I’ll stop now. The point is a lightweight CMS such as WordPress could probably do any one of these things, with a bit of work. But it isn’t designed to do anywhere near all of them. And each time it’s changed to do one more of these things the more it is moved away from its core architecture and it gets closer to a point of paralysis, where nothing functions well anymore because no part of it is doing what it was designed to do. A bit like the TCL example.

Looking back

Reviewing these two examples, it’s clear that the lightweight systems became, or would become, very costly when they were pushed beyond their initial expectations. In both cases the corresponding heavyweight systems came with their own (heavy) management structure, but that management structure ensures lower running costs.

In the Hibernate example our software maintained its architecture after we’d made our performance change; anyone looking at this new code would be able to rely on previous knowledge to understand what was going on. By contrast, anyone coming fresh to a snippet of old TCL code would be starting from scratch, regardless of how much of the other TCL code they’d seen.

Similarly, the large-scale content management system at GU is internally consistent, despite its vast range of features and functionality. Once someone has learnt the principles (which, admittedly, are non-trivial) they can get to work on pretty much any part of it. Pushing WordPress to do that would have created a monster.

Lightweight systems take the management away from you. And that’s ideal, as long you don’t need that manageability.

Most recently at Guardian Unlimited our (previous) homepage had clever layout rules whose logic sometimes overrode the content that editors entered. The result was that editors were often confused that the page didn’t render according to what they had put in the system. The clever layout rules had been devised by close collaboration with the original editors and graphic designers — the expert users. They reasoned that they didn’t want anyone to unbalance the layout by entering inappropriate combinations of text and images.

But these layout rules had been forgotten over time, hence the confusion years later. Consequently the tech team was often called up regarding a supposed bug (“I’ve put this image in but it doesn’t appear”), and effort was expended only to discover that it was in fact a feature — much to the caller’s amazement. Our micro-lesson there was to give the end users the freedom they naturally expected, including the freedom to decide for themselves what was a balanced layout. If what they produced was unbalanced then the designers would steer them back in the right direction — a much more human corrective.

Those clever layout rules were an anti-feature. They were additional functionality that actually made users’ lives worse. Eventually we removed them.

Anti-features happen in highly experienced mega-corporations, too. In November 2006 Joel O’Software started what became known as “the Windows shutdown crapfest”. He compared the three shutdown options of Apple’s OS X with the astonishing nine or more shutdown options of Microsoft’s Windows Vista. Not only is that confusing for the user, but it was also incredibly painful to develop — Moishe Letvinn, a former Microsoft engineer, tells the sorry story.

But at least the Vista shutdown anti-feature made the problem visible. In the layout logic example, and others I know of, there is silent intelligence at work that leads to confusion and frustration without giving the user any visibility at all of what’s going on.

Anti-features are time-consuming to build in, because any features are time-consuming to build in. But anti-features also consume additional time in their removal.

One way to prevent anti-features is to help the end user determine the long-term consequences of what they want. Of course, you’d hope they’d think of that themselves, but you can’t avoid your responsibilities to the project as a whole if you see something that others have missed.

Another way is to adhere even more fervently to the Agile mantras of delivering early (before every last sub-feature is there), keeping it simple, and focusing ruthlessly on only the highest value work. This way we deliver first a front page without clever corrective layout logic, or one or two shutdown options only, and consider further enhancements later if we find we need them. Suggesting and doing this is easier said than done, of course, but if everyone trusts each other to listen honestly to what they have to say then it’s more likely the decisions made will be the best ones.

Meanwhile we can at least ask ourselves each time: Is this a feature or an anti-feature?

It is, however, as relevant to software as it is to building (and probably to many other areas of life). The constant pressure to deliver means there is very rarely the opportunity to go back and improve a nasty historical workaround which is causing problems today. However, there are some strategies we might consider…

1. Automated testing

If the nasty programmatic stickytape is relatively small then a comprehensive suite of automatic tests should enable you to make the change relatively safely and painlessly. Of course, you need to have built up that suite of tests in the first place. And the “relatively small” caveat is important, because only then can the software people fit the replacement activty into their daily work without disrupting any schedules. If there’s no external change then this is an example of refactoring.

If the nasty workaround isn’t so small, but really does need replacing, then a much more concerted effort is needed. A plan needs to be carefully devised which allows slow piecemeal replacement. The point of the plan should be to take baby steps towards the end goal; each step should also leave the system in a stable state, because you never know when you might have to delay implementing any subsequent step. However, if everyone in the team knows the plan then the end goal can be achieved with minimum disruption to the business’s schedules.

2. Act like a trusted professional

I find trust and transparency is an increasing part of the software projects I’m involved with. Part of this is that the technical people want to give their customers options, and are keen to explain the pros and cons, so an informed decision can be made.

However, while this is usually excellent, there can be times when it is too much or undesirable. Stakeholders don’t necessarily want to be given options for every single thing, and sometimes it’s right for a technologist to make an executive decision without referring back — because they’re a professional, and because they are trusted (and employed) to be professional. In this regard it’s sometimes the responsibility of a technologist not even to entertain the possibility of a temporary workaround knowing it will become permanent.

The decisions I’ve been close to in this regard tend to be architecture or technology decisions. For example, in my current work we have, roughly speaking, legacy technologies and modern technologies. Sometimes a feature requirement would arise which is quicker to implement in the legacy technology — but of course we knew it would be more costly in the long run. If the difference in timescales was not wildly divergent then I’d encourage my team to choose the modern technology and not to offer up a choice. These days I need to do that much less, because by building up that base of modern technology it’s now easier to expand and extend that for future features. By making hard decisions in our professional capacity earlier on we’ve made it easier to make the right decisions in the future.

3. Bite the bullet

A third way to remove the temporary workaround is to just be honest about the work involved and the cost to the business. Easier said than done, but an open and frank discussion followed by adjustment and agreement will ensure the issue is examined properly and supported more widely.

One example I remember is a particularly troublesome database table. Originally conceived as an optimisation with a few hundred rows, it grew over time to be a real albatross, slowing both development and production work, and running to over 19 million rows. However, we couldn’t replace it without a lot of effort, because its tentacles were everywhere; we needed to schedule real time for it, and that would mean less time for other, more visible, work.

But when we presented the plan, two things sugared the pill. First, we had long cited the table as the reason for previous work taking longer than anyone would like, so the cost of its presence was already felt tangibly. Second, we ensured our plan was broken down using the “baby steps” approach outlined above — it would take a long time, but it would take out no more than 10% of our resources at any moment, and we could always suspend work for a period if absolutely necessary. The plan won support, was executed, and after several months our DBA typed the SQL “drop” command and we popped a bottle of champagne.

Meanwhile, back in the kitchen…

Of course, all that is assuming the temporary workaround really does need to be replaced. In my friends’ kitchen they are actually quite happy with the drawer dividers they’ve requisitioned as a spicerack. Aside from anything else, it provokes conversation and allows them to talk about their Azerbaijani builder’s many philosophies. He seems to have so many pithy sayings — all of which begin “As we say in Azerbaijan…” — that they’re beginning to suspect he may be making them up as he goes along. Still, if he does have to make a hasty exit from the building trade there may be an opening for him in the software industry.

So I won’t repeat what they’ve said. Instead, I want to point to one tiny new feature which I’m unfeasibly excited about. It’s series. You know series. They’re those things which appear regularly in a paper or on a website.

In our redesigned sites we’ve introduced series as a specific and distinct concept. For example, Bad Science is a weekly series by Ben Goldacre on pseudo-science. Ask Jack is a series in which Jack Schofield helps people with their computer problems. In the old world these articles would be gathered into their own sections and left at that. But a series is a little bit more than this. For one thing when you land on an article in a series it’s nice to know that it is one of a family. We now flag that at the side of the piece. For another thing, you should be able to navigate to the next and previous articles in the series. Again, this series-specific navigation appears with each piece.

It’s a very small thing, but to me it shows something fundamental and important: the software we’re producing really does match the way our writers and editors think. It is, for those in the know, a result of domain driven design. For everyone else, it’s simply software that does what you think it should do.