niksilver.com

Today we’ve launched three new sites on Guardian Unlimited. Well, more correctly, we’ve redesigned and rebuilt our Science, Technology and Environment sites. Working for a media organisation, I can rely on people much more articulate than me to tell the stories better — see Alok Jha on Science, Alison Benjamin on Environment and (with more technical info than you can shake a memory stick at) Bobbie Johnson on Technology.

So I won’t repeat what they’ve said. Instead, I want to point to one tiny new feature which I’m unfeasibly excited about. It’s series. You know series. They’re those things which appear regularly in a paper or on a website.

In our redesigned sites we’ve introduced series as a specific and distinct concept. For example, Bad Science is a weekly series by Ben Goldacre on pseudo-science. Ask Jack is a series in which Jack Schofield helps people with their computer problems. In the old world these articles would be gathered into their own sections and left at that. But a series is a little bit more than this. For one thing when you land on an article in a series it’s nice to know that it is one of a family. We now flag that at the side of the piece. For another thing, you should be able to navigate to the next and previous articles in the series. Again, this series-specific navigation appears with each piece.

It’s a very small thing, but to me it shows something fundamental and important: the software we’re producing really does match the way our writers and editors think. It is, for those in the know, a result of domain driven design. For everyone else, it’s simply software that does what you think it should do.

“Quick and dirty” is a weapon that’s often deployed by people for getting themselves out of a tight spot. But like any weapon, in the wrong hands it can backfire badly. Technical people use techie quick and dirty solutions in their work. Those people we like to work so closely with — our clients — use their own quick and dirty solutions in their work. But what happens when these people come together? What happens when it’s not the developers or sysadmins who suggest their own quick and dirty solution, but when it’s suggested by our customers on our behalf?

When it’s the client who suggests we deploy a quick and dirty solution, there’s a few things that we need to consider…

1. What does “dirty” mean?

There’s no doubting what “quick” means. But what does “dirty” mean? It’s something that makes the technicians itchy, so it’s worth understanding why.

A dirty solution might suggest one that’s inelegant, that doesn’t conform to a techie’s idea of a good design. That might be true, but it’s probably not the core of the matter. Any decent techie who works in the real world doesn’t advocate elegant design for its own sake. They know that most of their designs are far from perfect, and in fact they probably pride themselves in maintaining a pragmatic balance between the ideal and the expedient — this, then, won’t necessarily make a techie uncomfortable.

So “dirty” doesn’t mean “inelegant”. I think in this context it means something which incurs technical debt. That is, it’s a technical shortcut for which we are going to pay a rising cost as time goes on. That’s what causes the discomfort — the knowledge that implementing this will be making the techie’s life, and the life of their client, difficult in ways that cannot be really understood until it’s too late. It might happen also to be a inelegant solution, but that’s by-the-by. The important thing is that “dirty” implies something which carries a cost.

Of course, the cost of dirty has the benefit of quick. And as we go on, seeing this in cost/benefit terms is actually very informative…

2. Who gets what?

If you’re my customer and I’m your techie, then there’s a significant and unfortunate thing about quick and dirty that I need to tell you: You get the quick, but I get stuck with the dirty. Or to put it another way: the cost and the benefit are not felt by the same person.

There are times and circumstances when this is not the case. If an individual sysadmin is making a judgement call about their own work, say, then they may choose a quick and dirty solution. And if we are considering an in-house software team with in-house customers, then the company as a whole will both benefit from the quick and pay the cost of the dirty.

But these examples don’t tell the whole tale. If our individual sysadmin is actually one member of a systems team, and has responsibilities to the team, then it’s their colleagues who are going to be sharing the cost — and in fact it’s not entirely correct to have called them an “individual sysadmin” in the first place. And while it’s true to say our example company reaps the benefit and pays the cost, it’s probably not the case that “the company” has made an informed and conscious decision about the cost and benefit. That’s because it’s very unlikely that such a decision will have been escalated to the single body responsible for both the cost and the benefit — the board, or the owner, or the managing director.

In the end, we must recognise that the one who reaps the benefit is probably not the one who’ll be paying the cost.

3. Who decides what?

The “who gets what?” question leads in to a more general point: experts must be trusted to make judgements in their own areas.

So just as a techie should not tell a customer how to do their job, so the customer should not tell the techie how to do theirs. And in particular, it’s not up to the customer to decide whether the technical solution should be a dirty one.

Of course, we should always feel free to make suggestions; no-one is so big that they shouldn’t be open to ideas. It’s fair and reasonable for the customer to ask the techie if they can do it quicker… and for that the techie would need to demonstrate some creative thinking. The customer may ask if there are better solutions… and the techie would need to do some research. And the customer might even suggest a specific technical solution they’d heard of… and the techie would have a responsibility to give an honest and informed opinion. But if someone is supposed to be the expert on technical issues then they do need to be free to make judgement calls on technical issues, including which solutions are and are not appropriate. That shouldn’t be a worry to the customer; as mentioned above, good technical people tend to take pride in their pragmatism.

And in fact, this should be good for everyone. By trusting everyone use their expertise we should get the best solutions.

4. How long does it last?

One final point on the cost/benefit view of quick and dirty: Quick lasts only a short time, but dirty lasts forever.

More precisely, the benefit of quick lasts from the time the quick thing is released to the time the “proper” thing would have been released; and the cost of dirty also lasts from the time the quick thing is released, but it goes on until the time the software is rewritten or deleted.

I’d say that unless software is a total failure it tends to last a very long time, and usually much longer than anyone ever would have wished. This is why the term “legacy software” is heard so often — if software was generally replaced or removed in good time then we’d never have needed to invent that phrase. And the fact that the dirty bits outstay their welcome means that the technical debt they incur is paid, and paid, and paid again… certainly long after the shine of being early to market has worn off.

But it’s not all bad

I don’t want to give the impression that quick and dirty is inherently bad. It’s not. Nor is everything quick necessarily dirty. Far from it; some things are just plain quick.

But if we’re talking about quick and dirty then it helps to think in cost/benefit terms. We need to be very clear about who’s paying the cost and who’s reaping the benefit, and exactly what that cost and its benefit is. If we can understand that, and if we can respect each other’s expertise in making these decisions, then we can be more certain that we’re making the best decisions a better proportion of the time for the long term benefit of everyone.

We’ve just launched the single most visible, most complex, and most trafficked page in the entire network of Guardian Unlimited sites — the Guardian Unlimited front. And in so doing we’ve revealed more of a new design that’s due to roll out over the next few months.

There are some really lovely editorial and commercial features in there, but I can’t give the whole game away now. Those tricks and subtleties will reveal themselves over the next few days, weeks and months as the news agenda and business opportunities unfold. However, I would strongly recommending looking at Emily Bell’s piece which explains what we’re doing, and Mark Porter’s background on the design.

Since this is a technology blog I want to focus on just a couple of technical areas that I’ve been tracking — templating and layout, and some of the issues we’ve had there.

Keeping it simple with Velocity

Like the GU Travel site, perhaps our highest profile release in recent months, the templating was achieved with Velocity. This is no secret; Velocity was named as one of our technologies when we were recruiting for client-side developers earlier this year.I’ve noted with raised eyebrows the Velocity versus Freemarker skrimishes that have blown up on the Net from time to time — they have been both lengthy and deeply felt. I can’t honestly say that we ran a detailed and open comparison when we chose a templating framework, but we consciously chose Velocity because it enforces simplicity. I once recommended someone to the Velocity documentation page, and one of our developers piped up “That’s what I like about Velocity, it has a documentation page“. Admittedly, it’s a very long page, but it’s still one page. That’s a sign of its simplicity.

But maintaining simplicity is difficult, and it’s too easy when faced with basic tools and a complex problem to create hacks and workarounds. One of our warning signs of things going skew-whiff was when our Velocity templates were becoming difficult to understand. I know a few people who would regard such things as the norm, that workarounds are par for the course for a jobbing techie. But complexity of the solution should be proportional to the complexity of the problem, and rendering data in templates shouldn’t be complex unless the data is complex. Also, one of the reasons behind the strategic decision to keep the development in-house was to ensure quality remains high throughout the software stack — we can resolve problems at source, not just where we have access.

In the case of complicated Velocity templates I was at pains to emphasise to our client-side team that they should expect simple, intuitive, relevant information from the back-end systems, and they should use this to ensure their templates are easy to read — if they weren’t getting this then they should tell the back-end developers. Conversely the back-end developers considered it their duty to provide simple, intuitive, relevant information to the client-side team, but needed to be guided as to exactly what this meant.

Consequently, subtle changes in the way information was presented, or accessed, made big differences. The back-end team ensured not only that the domain model was adjusted and expanded to ensure simple templates, they also built a number of tools to ensure the work of templating itself was more manageable.

In the end the Velocity behind the Guardian Unlimited front does remains complex. But it’s complex only as far as the visual design itself is complex. That feels appropriate to me.

CSS: Cascading scalability snafus

Because there’s an awful lot of content on the GU front there’s an awful lot of styling. And that’s brought with it its own problems. An Agile process encourages change, and as the work came together the design and the styling necessarily evolved. However, while change can be handled easily in software (thank you, JUnit and Eclipse) that’s not really the case with CSS — there’s no reliable refactoring tool that I know of.

I’ve previously found Factor CSS which reduced a 300 line file I gave it to 40 lines. But as one of our client-side team stressed to me, “Refactoring CSS is about making it more manageable, not shorter”. IntelliJ IDEA does better. It allows renaming of IDs, class names, etc, which is a huge jump forward, but far from the current state of refactoring in object-oriented programming. So refactoring CSS is important to us because of that manageability: this is software we’re writing for the long term.

Thus a short time ago we embarked on a manual refactoring exercise which rounded up all those tiny changes that had crept in throughout the Agile design process. When we began we had a single global CSS file of several thousand lines. That horrified me, though I suspect it wouldn’t be shocking to those who work on this kind of thing full time. There is debate within the team as to whether this is to be expected — “CSS doesn’t scale,” / “Yes it does, but we’re still on the learning curve”.

Either way, we’ve now split it out across several more manageable files and removed a lot of duplication and redundancy. No-one thinks we’ve achieved an ideal, but we think we’ve done what’s pragmatic for now. The team has achieved this partly through dogged hard work, and partly with a bit of innovation. The hard work needs little explanation — it’s work, and it’s hard, and it’s part of any job, although in this case it does have the virtue of rewarding you with incremental results at every step of the way. The innovation is in the way we link CSS templates to the on-page components that they refer to, so that each component pulls in its own CSS if necessary, and only if necessary. This ensures we can isolate component-specific CSS quickly and effectively, and therefore be more confident in our refactoring.

However, I’m left with the feeling that web design is a good way behind back-end software development when it comes to management.

Separating presentation and content

Of course, separation of presentation and content is second nature today, if rare in older systems. I noted with interest that when Yahoo! News UK relaunched recently one of the innovations listed was “proper separation of presentation and content for the first time”. Yahoo! News UK and GU clearly have more in common than just what’s on the surface.

But while the fact of presention and content separation is well-established, the two sides must communicate and how this is done is much less certain. To give a particular example, when we display an article the content consists of obvious things such as article body, headline, byline, etc. Our early adherence to domain driven design (DDD) led us to ensure that this content — and only this content — was visible to the Velocity templates. This sounds fine in theory, but in practice wasn’t good enough: the visual design of the page also depended on knowing non-domain information such as where the 200-word point was, or the dimensions of the associated image.

These things are part of the presentation logic, so need to be kept separate from the domain logic. On the other hand, they also need to be kept separate from Velocity, because that’s a templating framework, not a programming framework.

For these reasons passing in “fake” articles with extra (non-domain) information violated the architectural principles of DDD and lost us its advantages. We also flirted briefly with creating and exposing objects such a DecoratedArticle (using the decorator pattern) but this didn’t pass the intuition test. A DecoratedArticle is not an intuitive thing to anyone except an experienced software developer.

It took some time to evolve a solution that ticked the boxes for domain driven design, simplicity, intuitiveness and manageability. It involved small alterations in our templating patterns, among other things.

The GU front in retrospect

The work to date on the GU front has been far from trivial — not only because of the issues above, but also because of things I’ve not talked about such as performance and the software our editors need to keep updating it.

Along the way we’ve tackled new and difficult technical issues. You might think it’s odd that such issues weren’t resolved last year when we were working on the Travel site which is really part of the same system. But, as I said before, the GU front is probably the single most complex thing we could release. It’s only when you force yourself to confront the most difficult issues that you are forced to see root causes and address them at source. This is particularly so as there is much more work to come. As Emily says, “our view is that for a more technically robust site, which will be a platform for far more rapid developments in the future, an iterative approach is the best”. So we’ve made a good decision to face the difficult stuff sooner rather than later. There will be more technical challenges to tackle in the months ahead. But for now I can say we’ve resolved some really core issues in ways that should hold us in good stead for a long time to come.