Category: Guardian.co.uk

Guardian Unlimited’s new look: Some background on templating

We’ve just launched the single most visible, most complex, and most trafficked page in the entire network of Guardian Unlimited sites — the Guardian Unlimited front. And in so doing we’ve revealed more of a new design that’s due to roll out over the next few months.

There are some really lovely editorial and commercial features in there, but I can’t give the whole game away now. Those tricks and subtleties will reveal themselves over the next few days, weeks and months as the news agenda and business opportunities unfold. However, I would strongly recommending looking at Emily Bell’s piece which explains what we’re doing, and Mark Porter’s background on the design.

Since this is a technology blog I want to focus on just a couple of technical areas that I’ve been tracking — templating and layout, and some of the issues we’ve had there.

Velocity hello world snapshotKeeping it simple with Velocity

Like the GU Travel site, perhaps our highest profile release in recent months, the templating was achieved with Velocity. This is no secret; Velocity was named as one of our technologies when we were recruiting for client-side developers earlier this year.I’ve noted with raised eyebrows the Velocity versus Freemarker skrimishes that have blown up on the Net from time to time — they have been both lengthy and deeply felt. I can’t honestly say that we ran a detailed and open comparison when we chose a templating framework, but we consciously chose Velocity because it enforces simplicity. I once recommended someone to the Velocity documentation page, and one of our developers piped up “That’s what I like about Velocity, it has a documentation page“. Admittedly, it’s a very long page, but it’s still one page. That’s a sign of its simplicity.

But maintaining simplicity is difficult, and it’s too easy when faced with basic tools and a complex problem to create hacks and workarounds. One of our warning signs of things going skew-whiff was when our Velocity templates were becoming difficult to understand. I know a few people who would regard such things as the norm, that workarounds are par for the course for a jobbing techie. But complexity of the solution should be proportional to the complexity of the problem, and rendering data in templates shouldn’t be complex unless the data is complex. Also, one of the reasons behind the strategic decision to keep the development in-house was to ensure quality remains high throughout the software stack — we can resolve problems at source, not just where we have access.

In the case of complicated Velocity templates I was at pains to emphasise to our client-side team that they should expect simple, intuitive, relevant information from the back-end systems, and they should use this to ensure their templates are easy to read — if they weren’t getting this then they should tell the back-end developers. Conversely the back-end developers considered it their duty to provide simple, intuitive, relevant information to the client-side team, but needed to be guided as to exactly what this meant.

Consequently, subtle changes in the way information was presented, or accessed, made big differences. The back-end team ensured not only that the domain model was adjusted and expanded to ensure simple templates, they also built a number of tools to ensure the work of templating itself was more manageable.

In the end the Velocity behind the Guardian Unlimited front does remains complex. But it’s complex only as far as the visual design itself is complex. That feels appropriate to me.

Anatomy of CSS rule, thanks to josephtateCSS: Cascading scalability snafus

Because there’s an awful lot of content on the GU front there’s an awful lot of styling. And that’s brought with it its own problems. An Agile process encourages change, and as the work came together the design and the styling necessarily evolved. However, while change can be handled easily in software (thank you, JUnit and Eclipse) that’s not really the case with CSS — there’s no reliable refactoring tool that I know of.

I’ve previously found Factor CSS which reduced a 300 line file I gave it to 40 lines. But as one of our client-side team stressed to me, “Refactoring CSS is about making it more manageable, not shorter”. IntelliJ IDEA does better. It allows renaming of IDs, class names, etc, which is a huge jump forward, but far from the current state of refactoring in object-oriented programming. So refactoring CSS is important to us because of that manageability: this is software we’re writing for the long term.

CSS editing with IntelliJ IDEAThus a short time ago we embarked on a manual refactoring exercise which rounded up all those tiny changes that had crept in throughout the Agile design process. When we began we had a single global CSS file of several thousand lines. That horrified me, though I suspect it wouldn’t be shocking to those who work on this kind of thing full time. There is debate within the team as to whether this is to be expected — “CSS doesn’t scale,” / “Yes it does, but we’re still on the learning curve”.

Either way, we’ve now split it out across several more manageable files and removed a lot of duplication and redundancy. No-one thinks we’ve achieved an ideal, but we think we’ve done what’s pragmatic for now. The team has achieved this partly through dogged hard work, and partly with a bit of innovation. The hard work needs little explanation — it’s work, and it’s hard, and it’s part of any job, although in this case it does have the virtue of rewarding you with incremental results at every step of the way. The innovation is in the way we link CSS templates to the on-page components that they refer to, so that each component pulls in its own CSS if necessary, and only if necessary. This ensures we can isolate component-specific CSS quickly and effectively, and therefore be more confident in our refactoring.

However, I’m left with the feeling that web design is a good way behind back-end software development when it comes to management.

Domain Driven Design book coverSeparating presentation and content

Of course, separation of presentation and content is second nature today, if rare in older systems. I noted with interest that when Yahoo! News UK relaunched recently one of the innovations listed was “proper separation of presentation and content for the first time”. Yahoo! News UK and GU clearly have more in common than just what’s on the surface.

But while the fact of presention and content separation is well-established, the two sides must communicate and how this is done is much less certain. To give a particular example, when we display an article the content consists of obvious things such as article body, headline, byline, etc. Our early adherence to domain driven design (DDD) led us to ensure that this content — and only this content — was visible to the Velocity templates. This sounds fine in theory, but in practice wasn’t good enough: the visual design of the page also depended on knowing non-domain information such as where the 200-word point was, or the dimensions of the associated image.

These things are part of the presentation logic, so need to be kept separate from the domain logic. On the other hand, they also need to be kept separate from Velocity, because that’s a templating framework, not a programming framework.

For these reasons passing in “fake” articles with extra (non-domain) information violated the architectural principles of DDD and lost us its advantages. We also flirted briefly with creating and exposing objects such a DecoratedArticle (using the decorator pattern) but this didn’t pass the intuition test. A DecoratedArticle is not an intuitive thing to anyone except an experienced software developer.

It took some time to evolve a solution that ticked the boxes for domain driven design, simplicity, intuitiveness and manageability. It involved small alterations in our templating patterns, among other things.

The GU front in retrospect

The work to date on the GU front has been far from trivial — not only because of the issues above, but also because of things I’ve not talked about such as performance and the software our editors need to keep updating it.

Along the way we’ve tackled new and difficult technical issues. You might think it’s odd that such issues weren’t resolved last year when we were working on the Travel site which is really part of the same system. But, as I said before, the GU front is probably the single most complex thing we could release. It’s only when you force yourself to confront the most difficult issues that you are forced to see root causes and address them at source. This is particularly so as there is much more work to come. As Emily says, “our view is that for a more technically robust site, which will be a platform for far more rapid developments in the future, an iterative approach is the best”. So we’ve made a good decision to face the difficult stuff sooner rather than later. There will be more technical challenges to tackle in the months ahead. But for now I can say we’ve resolved some really core issues in ways that should hold us in good stead for a long time to come.

Want a job in video or audio?

My colleague Neil McIntosh writes on his blog:

I’m pleased to report this blog has had some success on the recruitment front, so here goes again… and this time, it’s for two of my teams.

Indeed, one of the new members of our expanding software team found their way to us through one of Neil’s previous (and snappy) posts. As he remarked at the time, “Possibly the most profitable eleven words I’ve ever written.” (If so, only because your other writing is invaluable, Neil, only because of that…)

By way of returning the favour, I’m happy to say our multimedia capability is also expanding significantly, and we have eight new positions covering four distinct roles: Head of Audio, Head of Video, Audio Producers and Video Producers.

Take a look at Neil’s blog for some background. It’s probably one of the most exciting A/V opportunities around at the moment, and certainly a terrific bunch of people to be working with.

What management buy-in really means

We work using agile development processes, which obviously needs the buy-in of internal users and project sponsors. But this jumped out at me when I read it. It’s from Carolyn McCall, Chief Executive of Guardian Media Group, which owns the company I work for. She was announcing a £15m investment in our digital business, and it was reported thus:

“What we’ve done so far is our own version of web 1.0, but we want to continue to web 2.0 and what comes after that,” Ms McCall added. “We need to be agile and ready to change.”

I suspect that phrase, “We need to be agile and ready to change”, is not a coincidence. You can trace a strong lineage between the agile development work going on in Guardian Unlimited and that little sentence in a speech to the OPA. Let me tell you how I think it got there.

A story of a sentence

A long time ago we began using agile processes in the GU tech team. It was a new way of working, but it seemed to address a lot of our problems — actually, the usual ones you get in most software organisations, such as knowledge silos, lack of flexibility, impossible deadlines, and so on. Changing what you know is always a tough choice, but the GU management team were very supportive of the move to agile.

In particular, our boss at GU at the time, Simon Waldman, was very interested in its implementation and evolution. (Simon’s since moved to GMG.) Over the months and years he enquired, queried, provoked, but was never less than encouraging and constructive. (At one memorable annual review to the GU staff Simon devoted an entire slide to explain how we had stealthily removed a problematic database table. Our DBA was thrilled; I have no idea what the attendant sales people and journalists made of it.)

And agile clearly worked for GU. One notable change was that planning meetings took on much more of a business focus. Far less about technical dependencies, much more on what functionality we wanted to release when.

And then we began we work on our new Travel site (released in November). It featured a new design and new commercial opportunties. More people around the company needed to scrutinise this as it started up, because it was a pretty big project and we had a much greater responsibility to be rigorous and transparent in our delivery. We saw agile methods as being critical to the work’s success. It would allow us to delay decisions to the latest possible moment and therefore produce something that was much more relevant — both editorially and commercially — to our audience and clients.

I therefore spent some time explaining agile to various company directors… but not as much as you might expect, because clearly the principles and practices had been discussed and understood well before. I ended up having the kind of conversations about agile development with non-technical senior managers that I wish I could have had with more technical job candidates. Word was getting around. The business people scrutinising the work on the Travel site understood the process’s business benefits, and they understood how the low-level practices would provide those, and they knew those practices were working successfully within GU at the time. And indeed, the project was blessed by the the company directors, including Simon and our MD at the time… Carolyn McCall.

You can see how that word “agile” has been thrown around a lot inside the company, and was — at least then — strongly associated with Guardian Unlimited and therefore its innovation (which I’d like to think is almost synonymous with “daily work” — ahem), and it’s stuck.

Now I can’t claim that Carolyn knows what, say, test-driven development really is. I suspect she has higher level things to think about. But if it came up, I think she’d understand our practices such as refactoring (making lots of tiny internal restructurings to produce a smoother, more manageable system), continuous integration (ensuring our changes are constantly integrated into the day-to-day work, not siloed) and retrospectives (frequent reviews and actions to improve). And, come to think of it, TDD, which is really about supporting and enabling change.

It seems Carolyn understands “agile” in a way that I wish more technical people would. It’s about change, and about supporting change (“We need to be agile and ready to change”).

The hard work starts now

And what of the future? Well, agile development helps us deliver responsibly to the business — and if someone invests £15m in you, then you really need to deliver responsibily. Again, that kind of investment only happens if there’s confidence in your ability to deliver. Can we offer that confidence? Well, just this week I was in a meeting with Tom Turcan, who is our Head of Digital Media Development. The conversation went something like this. Note that development for this particular thing is already about 30% of the way through:

Expert user: “…So in summary we’ve decided that features A, B and C aren’t that important after all, and we’ve replaced them with features J and K.”

Tom: “It looks like A, B, C total 7 ideal days, and J and K total 8.5 ideal days. So that’s a net increase of 1.5 ideal days. Is that right?”

Expert user: “Er, yes.”

Tom: “So we can’t really allow that to happen. We can make changes to make the work less, or keep it at the same level, but we mustn’t start increasing it. Can you find 1.5 days’ worth of features to remove?”

Expert user: “Yes, I should think so.”

Tom: “Okay, we’ll make a note to see what you’ve decided next week.”

We couldn’t have had that conversation in a more waterfall environment. By the time we’d have reached this point the groundwork for A, B and C — the database tables, the DAO layer, etc — would have been done, and it would have been wasted effort. Instead our expert user is able to make decisions later, and Tom is able keep to keep the work under control, giving the rest of the business the confidence it needs. Our agile processes enable that to happen. And all that is known by our developers, and by our managers, and that same confidence is shared by our group Chief Executive.

Postscript

By the way, there are other things going on this week in and around Guardian Unlimited. You might want to take a look at Jeff Jarvis’s commentary on Alan Rusbridger‘s commitment to digital and his interest in being flexible. Also, my esteemed colleague Neil McIntosh tells how GU saved a cat, and offers a word of advice to a researcher at News International.

As a friend said to me the other day, “I know when you’re busy at work — your blog goes quiet.” So back to work now.

Guardian Unlimited is hiring a Development Manager

If you’re interested in the kind of work we’re doing here at Guardian Unlimited, then you might be interested in joining us. We’re looking for a Development Manager. As an experiment I put an ad on Google Base (now expired). Here’s the text in full…

Guardian Unlimited is embarking on a concentrated effort to expand its Development Team over the next 6-12 months, and is seeking an experienced Development Manager to drive that process.

The ideal candidate will have excellent knowledge and experience of website operations, of managing a team of developers, and of hiring excellent new developers. Additionally, you will have experience of working in a fast-changing and pressured environment, with demonstrable expertise in managing change in development teams. Excellent planning and organising skills are also essential

Technically, the team practises agile development with Java, so you will need to demonstrate a deep understanding and affinity for both these areas. The underlying operating systems are Linux and Solaris, so expertise there would be an advantage.

The Guardian Unlimited workplace is very diverse with many different stakeholders, so it’s essential to be able to work with many different kinds of people and have excellent interpersonal skills.

This position will remain open until a suitable candidate is found, and ideally the successful candidate will start as soon as possible. This is an excellent opportunity to be part of an award-winning team on a product recognised for its innovation and quality.

To apply for this role you must be available to attend an interview in London, and to work permanently in the UK. If you are, and if you are interested in this role, please e-mail gujobs@guardianunlimited.co.uk and include (a) your current and desired salary, and (b) a copy of your CV.

We welcome applications from any individual regardless of ethnic origin, gender, disability, religious belief, sexual orientation or age. All applications will be considered on merit.

This role has come up because my job has expanded to encompass the whole software side, including project management and QA, so we’ve got a gap to fill.

A new Travel site, and four uses for tags

Here at Guardian Unlimited we’ve just launched our new Travel site. You’ll be able to read a lot about it in the industry press, but I just wanted to write a little what we’ve done from a technical-managerial point of view.

The project motivation is ideal

The main strands of our business are editorial, commercial and technical. But this is not an editorial project, it’s not a commercial project, and it’s not a technical project. This is something that’s been driven by all of us. Editorially we wanted to improve navigation and bring in more user-generated content. Commercially we wanted to offer more flexible advertising opportunities. And technically we wanted to exploit our content much more.

The new Travel site brings all of us together. One technical force was to exploit keywords, or tags, a bit like those on Del.icio.us and Flickr. This is a very flexible way of categorising content and hence allowing better use of it. Tags, in turn, allow us to generate much more relevant navigation and bring in related readers’ tips from our travel tips site, Been There. For example, if you’re reading about Croatia then you can also see a column headed “Readers’ top tips” which shows readers’ tips specifically about Croatia.

This is also the perfect opportunity to redesign, to better present that information. And just as we can present more relevant tips, so we can present more relevant advertising to our readers. And the redesign also allows us to shape the pages to better fit the ads — the page can close up around a small ad, and expand to comfortably fit a larger one.

Of course, there’s a lot of technical stuff involved, but a technical project is no good for the sake of it. Here, the three strands of our business were driving each other. An ideal scenario.

Aside from much better navigation and contextual advertising, and since this is a technical blog, here are four of the other (smaller) things we’ve used tags for…

There’s a page for every tag

Each tag has its own page! Here’s the page for San Francisco. And here’s the page for skiiing. And here’s one for food and drink trips.

In fact, we’ve got hundreds of tags in the system already — you can see our list of places and activities, for example. And now we’ve automatically got a page for each of them.

There’s an RSS feed for every tag

Okay, not a big leap technically, but terrific for our readers. Top right on each subject page is the RSS link for that subject. If you’re interested in an area of Travel then it’s odds on we write about it. And if we write about it then we’ve got an RSS feed for it. You can get your own personal skiiing feed from Guardian Unlimited.

We can combine tags

Tags also allow us to pull together even more really specific and relevant content. If you’re on the San Francisco page you can find out about food and drink in San Francisco. Or you can investigate short breaks in Portugal. Or water sports in the UK. Well, you get the idea.

Friendly URLs

Our URLs are usually of the form

http://travel.guardian.co.uk/budget/story/0,,991699,00.html

which is short but a bit obscure. So we’ve taken the opportunity to improve that for the new Travel site. The same article above now has the URL

http://travel.guardian.co.uk/article/2003/jul/05/watersportsholidays.budgettravel.boatingholidays

It’s longer, but we hope it’s also more intuitive. You can clearly see the what it is (an article), the date (5 July 2003), and what it’s about (water sports, budget travel, and boating holidays).

And it’s not just articles which have friendlier URLs. Here’s the URL for the Hamburg tag page:

http://travel.guardian.co.uk/tag/hamburg

and here’s the RSS feed for our Hamburg content:

http://travel.guardian.co.uk/tag/hamburg/rss

All much easier to deal with.

And one other thing…

Lots of this is automatic. But there’s one important part of tagging that’s not: the actual act of adding the tags. In theory we could use software to look at each of our articles and determine what it’s about, and so apply the tags. But that’s not going to give the best results for our readers. So we’ve made sure that all our articles have had their tags considered and chosen individually by someone who has actually read the article. That means the navigation should be spot on, and the advertising links are ideally targetted. There’s an awful lot that machines can do, but sometimes it needs a human being to add the magic.

Anyway, hope you like the site.

We’ve won a Webby!

How great to be able to make this the first post of this new blog. Guardian Unlimited has won a 2006 Webby — winners of the Newspaper category. We were nominated with illustrious names from world of US print, and it’s great to see our name among theirs. This is the second year we’ve won it, too, which is particularly wonderful, if slightly unsettling… it means there’s much more at stake over the next year.

It’s great to be part of such a star team. My colleague Sheila Pulham has more news and links on it (and it’s from her that I stole the unavoidable headline) over at the GU Newsblog.