niksilver.com

Agile comes with tons of literature on how to organise work at a very detailed level. How much of this treadmill is activity for activity’s sake, rituals, and religious manifestations of an ‘Agile Sub-Culture’ aimed at integrating a growing workforce? Can this relentless ‘heart beat’ and this esoteric jargon stifle innovation and alienate the very people Agile purports to help? Could more slack benefit a discipline which ultimately relies on people’s intelligence and creativity?

That’s how we set the scene a few days ago when I led a discussion among some peers from different companies entitled “The Agile treadmill: Command and control in disguise?” It was organised by IndigoBlue as part of their Second Wednesday series, under the Chatham House Rule. This is summary of what we discussed, with fictionalised initials for each participant.

In this article

• Origins. How the question arose.
• How useful is Agile process? It’s not all bad.
• Religious Agile, variable results. Examples to demonstrate that religious Agile is somewhat independent of project success.
• So what is Agile, anyway? To be best understood, agile working should be seen not from the development point of view.
• Seeing the vision, from top to bottom. The importance of all stakeholders seeing the whole picture.
• What did we learn? That religious Agile is independent of project success, that agile is a business methodology, and that the vision must be shared.

Origins

When people talk about how Agile they are I find this usually coincides with how rigorous their processes are. Some of the largest, most successful Agile projects I know have an exhausting hothouse atmosphere, where everyone’s role is tightly defined and cannot be deviated from.

Have we created a kind of Animal Farm, where a system designed to set us free has actually enslaved us? Does it make a difference if Agile has been introduced by developers or management? Is effective Agile synonymous with a hothouse environment?

Between us we had an awful lot of experience to help answer these questions.

How useful is Agile process?

There was a general agreement with TC that there is some real value in having standard processes and practices:

• Rewriting the rulebook every time you start a project or programme is wasteful if you can capture the processes and practices that are known to work;
• It’s easier to bring in external resources if well-known language and practices are being used in the team.

But practices can be unnecessary, and there was recognition that many practitioners have a near-religious attachment to them, and to particular ways of working. As people managing programmes of work we often don’t have the discipline to recognise what the right level of discipline is… and isn’t.

There was also concern that Agile created an alienating language. NA indicated his particular discomfort over the word “customer” because that was often not how people saw themselves or their relationship with the tech teams.

Religious Agile, variable results

So no-one was anti-process (nor anti-Agile process), and we know it has produced good results. Our own major rebuild and redesign at the Guardian is one example of that, with a very highly regimented process and an outcome that at least met the aims and expectations. There are plenty of others.

On the other hand we collectively had experience of strong-process projects which weren’t delivering.

FC talked about Agile projects in his organisation. Most of the smaller ones were fine; these were projects with just a handful of developers. But the company’s most significant piece of work was having serious problems. It was now in its third year — nothing necessarily wrong with that, we said, after all some projects are just very large. And what had been delivered so far? Not much. Ah, we see the problem. And yet, he said, to the people on the ground this was a successful Agile project, because they were doing all the right things: daily stand-ups, rigorous unit testing, prioritised iterations… all the paraphernalia of an Agile project. They couldn’t see there was a problem.

AS reported a similar experience. “When I walked into the organisation they proudly walked me through their very rigorous process. ‘But nine out of your ten projects are on Red’, I said, ‘What is that process doing for you?’”

So what is Agile, anyway?

“Is Agile a process?” asked AS, rhetorically, “No. Is there process in it? Yes.” Process helps a project fit into the overall programme, added FC.

BJ went a bit further: successful Agile is about business change, not about what happens in the software team. The Agile approach to a large project is to get just enough clarity to embark on it, which probably isn’t every last detail. The stakeholders need to have the confidence to deal with uncertainty, and that sufficient uncertainty will continuously be resolved as the project progresses.

[And with that I'm going to drop the capital A from Agile unless I'm referring to by-the-book developer-level Agile.]

RN then came in with his take on this. He works in a highly bureaucratic company where technology is driven by heavyweight specs, a focus by high level people on low level details, and delivered entirely by specialist (external) suppliers. Yet his board are continually wanting things delivered faster, to greater quality, and he sees agile as the solution to this. It means persuading his board to think about what they want at the right level — business value, not the contents of drop-down menus — and providing reassurance through clear, regular deliverables. It also means getting his suppliers to deliver in that manner. But all of that is entirely independent of unit tests, story cards, and so on.

I wondered if this meant a top-down mandate of agile was more successful than a bottom up one. AS clarified: it’s successful if it’s business-led, not management-led. LB linked this to Kotter’s 8 steps to successful change, and in particular the first step: there has to be a sense of urgency.

I confessed that our processes in the Guardian team today are substantially less religious (or rigorous) than they were during the time of our major rebuild programme. I used to be able to confidently say we are Agile and point to intensive pair programme, lots of story cards, and so on. Today pairing is inconsistent, automated unit tests aren’t used in non-critical satellite code, and teams don’t always work to iterations. I explained this changed because our business had changed. The emphasis is no longer on transforming our platform but now much more on innovation and on smaller projects with a much wider variety of stakeholders. This makes the control freak within me distinctly uncomfortable (AS had previously commented that in many organisations command-and-control provides confidence) but we generally deliver projects as intended and have happy stakeholders, so I wasn’t going to wade in to change it. BK commented that this seemed perfectly appropriate, and still looked like an agile business.

Seeing the vision, from top to bottom

Successful agile is something that needs to permeate the organisation, from top to bottom. It’s about delivering value, and the vision for what needs to be delivered must be seen and understood at all levels. Many of our experiences of projects going awry were also experiences of where the project vision wasn’t seen by the people doing the day-to-day work.

TC told how his IT department decided it was going to achieve CMMI Level 2. After two years they had accumulated 10 rooms’ worth of documentation, and their productivity had dropped like a stone. And yes, he could measure that drop in productivity, because the team essentially did nothing else for those two years than produce documentation. The board inevitably said: Never do that again. CMMI accreditation may be useful, but it’s got to be for a purpose.

FC talked about his troubled project in its third year. “I revisited the original outcomes document a short while ago, and I couldn’t see how the activities around me are ever going to achieve those outcomes. When I ask one of the team leads what they’ve delivered, and how much there is to go, I’m just waved in the direction of a vast wall of index cards.”

AS talked about a daily scrum he attended: a two-minute meeting in which the team lead got a task update from each team member while he stood with his back to the story card board. “I had to take him to one side and tell him that he needed to be constantly referring people to the project outcomes, not just getting task updates.”

He also told the story of a business manager who wanted a change rolled out midweek — unprecedented for the organisation, as it would involve taking the website down for two hours. This was flatly refused by the Head of Systems. When they discussed the matter the issues were this: the business manager could show that if the change rolled out in the desired slot the company was projected to make an extra £1m, and if the slot was missed there wouldn’t be another opportunity for six months; the Head of Systems, meanwhile, had a target of 99.9% uptime and two hours downtime would mean he’d forfeit his bonus. “Our business must organise itself so that we are all focused on the same goals.”

What did we learn?

You may ask what our learnings were. I can confidently say there were no learnings whatsoever because “learnings” isn’t a real word. But there were probably lots of lessons. Here are three key things I came away with.

1. Religious Agile, with an optional hothouse atmosphere, is entirely independent of a successful agile project. There are successful projects with and without religious Agile, and there are unsuccessful projects with and without religious Agile. This is underscored by…
2. Agile is about business change. It really only works when all the stakeholders focus on the same business-level goals, understand what their part in it is (and isn’t), can deal with uncertainty, and continually work to reduce that uncertainty exactly when necessary.
3. The business-level vision must be clear and permeate all levels. This is a corollary to the last point, but to me is significant enough to pull out. The people on the ground have to be able to relate all their activities to the business’s end goal, otherwise they will define success by other criteria — perhaps how well they adhere to a process — which are different from the actual success criteria.

Thanks to all the participants for making it such a stimulating morning.

In my spare time over the past few months I’ve been writing an app for my Palm Pre, and have finally got to point where I’m happy enough to release it. Palm Pre and Pixi apps run on Palm’s webOS operating system in HTML and Javascript. This is a summary of how I got on…

• Background
• Javascript++
• Development environment
• Execution sequence
• Callbacks and testing
• Events
• Releasing it

Background: A basic usability problem

I’ve always regarded Palm as the masters of productivity. David Allen based his Getting Things Done system on his use of a Palm Pilot. I stuck to my old Palm Vx as long as I could, as I lived almost entirely out of its task list application. I bought an iPod Touch when I had to ditch my old Palm, but while Apple makes beautiful products it doesn’t seem to care about productivity apps. (And that’s understandable — if you can look cool in Starbucks with a black polo neck sweater and a slim silver laptop, why should you care about productivity? Me, I don’t look cool in polo neck sweaters.) The iPod Touch and iPhone’s calendar app, for instance, only allowed a limited choice of event recurrence options; if you had a meeting which occurred on the 3rd Thursday of every month then that was just tough. There was no task list app as standard.

While I had migrated my task list to Remember The Milk (RTM) on my desktop, the Touch’s RTM app was for me a poor user experience. For one thing you had to navigate through several screens to get to your task list. Then you couldn’t see today’s and tomorrow’s tasks in one list — they were on separate screens you had to navigate back and forth between. And when you tapped to create a new task you couldn’t start typing “Get bread”, you had to first tap on the description field to select it — that’s just an annoying unnecessary extra user action. One lesson I learnt — from GUI Bloopers, I think — is understand what the user will primarily want to do and then absolutely minimise the work needed to do it. In a task list application I’d have thought having your tasks on the first screen might be obvious. Apparently not.

So I held out for the long-awaited Palm Pre to be my first smartphone. (And in the UK that was a very long awaiting.) Take a look at the Pre’s calendar app in this inevitably unnatural video run-through. Notice for one how the free time concertinas up so that you don’t miss knowing about appointments just because there’s a big gap before them. That’s an application made by people who really understand what their users need — or at least understand what I need.

Unfortunately, while the Pre does come with a very beautiful task list app it really is very basic. It doesn’t capture the ease of use of the Palm Pilot’s original (once again you have to drill down to actually see your tasks). And of course it was foolish to have hoped it would sync with Remember The Milk.

So, anyway, I wrote a task list app.

Javascript++

Palm’s webOS is based on Linux, and the basic programming environment is Javascript and HTML running in Webkit. Now I know what you’re thinking: isn’t that just what Steve Jobs announced when the iPhone first appeared, before the iPhone SDK existed, and wasn’t it unconvincing then? This is what he said:

We have been trying to come up with a solution to expand the capabilities of the iPhone so developers can write great apps for it, but keep the iPhone secure. And we’ve come up with a very. Sweet. Solution. Let me tell you about it. An innovative new way to create applications for mobile devices… it’s all based on the fact that we have the full Safari engine in the iPhone. [...] don’t worry about distribution, just put ‘em on an internet server. They’re easy to update, just update it on your server. They’re secure, and they run securely sandboxed on the iPhone. And guess what, there’s no SDK you need! You’ve got everything you need if you can write modern web apps…

That section of transcript came from Engadget’s live coverage, and they immediately added the following comment of their own:

Weeeeeaaaak

Well, don’t confuse that with what Palm has done, because there are three key differences. First, they’ve provided their own APIs which allow access to useful functions, including some system functions such as background alarms, network status, location, and so on. Second, you can package up your app into a single bundle and deploy it onto a device. Third, when an app runs it does so within a well-defined framework. So you can create scenes with their distinct logic and pass data between them, and the framework (called Mojo) handles the application lifecycle.

So Javascript is still the language, and HTML and CSS is still the rendering mechanism, but that’s just a language detail. You really are writing real apps on the actual device platform. It’s Javascript, Jim, but not as you probably know it.

As an example, here’s how my task list app exits from the scene in which someone has added or edited a task:

The Mojo.* Javascript classes are built in, and I’m using them to do a bit of logging and also to exit from this scene. The framework itself knows what the last scene was to return to, and the only thing I’ve added is some parameters to pass back.

Development environment

The webOS development environment and workflow is pretty good. It consists of

• Eclipse;
• The Aptana Javascript plugin;
• Sun’s VirtualBox; and
• The webOS SDK itself.

The SDK consists of

• Palm Pre and Palm Pixi images for VirtualBox (thus creating an emulator);
• Another Eclipse plugin which adds some webOS specific actions (deploy to emulator, create a new Mojo scene, etc);
• Some command line tools to perform actions such as tailing logs, creating an app package, deploying an app, etc;
• Graphical tools for resourcing monitoring and DOM inspection; and
• Sample apps to study.

What appeals to me about all this is Palm’s relaxed attitude to technology ownership. Write an IDE? Why go to the trouble when Eclipse already exists. Write an emulator? Run it in VirtualBox. And so on. Plus, the choice of Javascript as a development language ensures a very low bar to entry.

This relaxed (Palm would say “open”) attitude extends to deployment, too. Once I’ve packaged my Javascript app I can send it to the emulator or I can hook up my Palm Pre to my laptop with the USB cable and send the app down the line. No licencing, no cryptography, no UUIDs, just drop it on the device. That also means I can e-mail you the app package and you can drop it on your webOS device, too. As a result there’s been a lot of healthy activity around so-called “homebrew” apps — apps which aren’t available in the Palm App Catalog but can still be easily downloaded and installed.

I said the development environment was “pretty good”, but I have reservations. For one thing, the debugging and troubleshooting tools are a bit basic. But my more significant reservations are around use of Javascript as a development language. After coming from the nice, safe, statically-typed world of Java — having a dynamically-typed language with no safety nets of compile-time type-checking or intelligent auto-completion means I have to work harder. More of my thinking goes into typing rather than programming.

Also, Javascript is a language and platform that has its own quirks that stalled this non-expert developer, particularly when dealing with a large application…

Execution sequence

One thing I took a while to grasp was how Javascript handles its execution sequence. On the one hand we are told Javascript is single-threaded. On the other hand, it’s easy to have events fire out of order using setTimeout or in response to some user action. Eventually I worked out that Javascript will execute a sequence of code without interruption until it reaches completion, and then it will select another sequence which has queued up in the interim (from setTimeout, or a user action, or whatever). Overall this is really simplifies things, and it meant I could stop worrying about concurrency problems. (It also explained why I couldn’t write any code to successfully demonstrate a concurrency problem.)

However Palm realised that this is potentially bad, too, because a long-running routine can lock up the application — it will become totally unresponsive while the routine runs. So in order to preserve the user experience the webOS platform kills any routine that continues for more than 10 seconds without interruption. Unfortunately when this happens webOS doesn’t tell you (e.g. by logging a message) so you’re left to guess why your software has suddenly stopped. And the documentation of this “feature” is not terribly obvious, either. I spent quite a bit of time wondering what was going on when it happened to me. But eventually I discovered the cause, found myself in a small community of similarly-frustrated developers, and posted some sample code which showed the problem in action and also demonstrated that at least webOS doesn’t kill your whole application, just the long-running routine.

So I had to adapt my code to run in smaller, disjoint chunks. Fortunately by that stage I had become rather adept at dealing with…

Callbacks and testing

Most of the interesting Javascript extensions use callbacks, in particular XMLHttpRequest and the HTML5 SQL database API, which both return their results on callbacks. Initially I structured my code around XMLHttpRequest (more precisely, Prototype’s Ajax.Request wrapper) and I avoided asynchronous storage by putting all my data in cookies, which are accessed synchronously.

But eventually I felt I had to switch from cookies to SQL and convert my synchronous storage code to asynchronous code. Not only was that conversion quite painful, but trying to find a way of writing tests around so many, er, sequential asynchronous actions sent me down a lot of dead ends. I was using Yahoo!’s YUI test framework, which is quite elegant, but I had found the limit of its usefulness here.

In the end I devised a mechanism to break up my test cases’ code into asynchronous segments with pauses to aid sequencing. So a test would create a database table if it wasn’t already there, pause, clear down the table, pause, write some data into it, pause, and so on. Essentially I was pumping many of my tests full of race conditions (although it was fine in practice… generally) . What didn’t help was that to run the tests I now needed a browser which supported the HTML5 SQL database API, and on my Windows box that meant only Safari. Which, it turns out, is spectacularly disappointing: slow, resource-hungry, and oddly buggy. Test execution took three times as long, and it was only in a small part due to the little pauses I put in between database calls.

Events

But at least I settled on an approach. One area I never resolved satisfactorily was how best to handle events. Because of the way Javascript executes, or because of what my application does, there were a lot of “things that had to respond to other things”. I didn’t settle on a single way to manage those, and actually found myself using four techniques:

1. Writing event listeners. Various functions register with an object to show their interest in an event. The object adds each function to a list. When the event occurs the object calls the functions in turn. Here, the event happens and the functions are called synchronously.
2. A dummy function designed to be overwritten. This is a simplification of the first, where the object knows that it will have only one listener. When the event happens the object calls the dummy function. The interested code simply needs to replace the dummy function with its own implementation.
3. DOM events. The interested code listens for a custom event in the DOM. When the interesting thing happens the custom event is fired in the DOM for the interested code to pick up. Useful if the firing code and the listening code aren’t otherwise well connected.
4. Weird specialist class. Sometimes I found some actions depended on other things having happened first. For example, pushing out task changes depended on having first fetched a “timeline” ID from the Remember The Milk server, which in turn depended having an authentication token, which in turn depended on having an internet connection. To handle this I encapsulated the logic in a single class. It had a method called fire() which just went off and did the next thing it could.

Now having four ways to handle events isn’t necessarily wrong: arguably there are different kinds of events, and each kind may well benefit from its own design pattern. But I don’t have a handy classification of events, much less a mapping from them to appropriate design patterns. So perhaps here more than anywhere else in my code I felt I was guessing.

Releasing it

Undoubtedly my experience of writing a webOS application was a positive one: the very fact that I finished the project proves that. It is the most complete and release-quality piece of software I’ve developed in my own time. And now my smartphone has its missing app: Cloud Tasks. I can continue to live by my task list even when I’ve left my desk.

With some trepidation I’ve made the application available for others to download and use, too, in case they find it useful. The trepidation exists because it’s taken a substantial chunk of my time to develop and I really want to leave it now, and get on with the rest of my life. But I know there are plenty of people who use Remember The Milk in different ways, making use of features I’ve not implemented in the webOS client, and don’t want to implement, because I don’t use them myself. It’s not very community-minded to release something with minimal support. Instead I’ve chosen to release it under the GPL, so that others can develop it and any developments remain similarly freely available.

We’ll see what happens.

A couple of weeks ago I presented at the Agile Business Conference 2009. Thanks to all those at the DSDM Consortium who made it such a great event. The theme of the conference was how Agile can help in adversity (e.g. an economic downturn), and I provided a case study — title above — of how the digital technology team at the Guardian had responded to various organisational changes and challenges. Looking back over the last few years I found I’d learned many lessons from working with my friends and colleagues here. This is the summary of those lessons that I offered to the delegates at ABC09…

Exploit your successes

Our huge rebuild and redesign of guardian.co.uk was a successful project. Once it was done we could have slumped back, exhausted. But instead we used it as a springboard to start additional exciting (and arguably controversial) work. Primarily I’m thinking of the Open Platform, led by Matt McAlister. As I wrote at the time, getting that off the ground was a huge testament to the work and faith of many people in the company, particularly given that it’s a very intangible thing at launch.

Still, the Open Platform is built on top of the work in that earlier project, and in particular in that people recognised technology success and were prepared to back more.

Sustainable business needs sustainable technology

Much of our technology, particularly on the web side, is developed in-house. On the one hand that can seem expensive, but on the other hand it does mean we have the ability to take in the direction we see fit. A particular example is the cost of scaling guardian.co.uk. It does cost more money to serve more users, but it doesn’t cost as much as it might because we have control of the technology.

Broadly speaking since we own the technology we can steer it in line with business needs much more readily.

Right-levelled decision-making builds trust

Let’s make one thing clear first: “Right-level” is absolutely not an acceptable verb phrase, and I only used it because it fitted onto the slide better. Now, to the substance of this…

Any reasonably sizable projects are approved to go (or not go) by a group of directors and other senior staff, and they do this at a regular meeting on the basis of information provided by the technology team working with many people around the business. This information of course consists of timescales, costs and benefits, but also what other projects are in progress and waiting in the wings.

This may seem obvious, but the consequence is that in the event of heavy cost control (which is what happens when the economy takes a dive) that cost control is exactly in line with project expenditure, because it’s the same people who are making both kind of decisions. It should be said that this is supported by the Agile principle of delivering value, and delivering frequently, because any project can start demonstrating its value early.

Openness reduces costs and provides options

The Guardian has been open not only with the Open Platform, but also with our full content RSS feeds. Applications have been built on the latter simply because they are a known format and, with full content, have huge utility. By having open full content RSS feeds we have a means of others using our content, and if they (or we) choose that option there are no internal integration costs.

Transparency builds trust

We do a lot of internal reporting: what features we release, our current bug count, project costs, and so on. When times get tough people start asking what value you’re adding to the business, and where all that money that’s being spent on your team is actually going. Because of all our reporting it’s relatively easy to open up our books to the consultants with financial questions, and then we can have an intelligent debate about the value of that expenditure. But that’s fine, and something we tend to welcome, because it’s part of the culture that Agile fosters: within the team we debate every week the value of what we do — that’s what Agile planning is all about — and Agile retrospectives are all about seeking improvement.

And finally…

I recognise that saying all this may be a hostage to fortune: the credit crunch took hold only about a year ago, and we’re not out of the woods yet. But for now I can look back on some of the things that have gone well, and if we emerge from this economic downturn with only a few cuts and bruises then it will be in no small part due to honesty and openness creating trust, which in turn leads to healthier decision-making.