niksilver.com

There’s a post from Joel O’Software regarding measuring performance and productivity. He’s saying some good stuff about how these metrics don’t work, but I’d like to balance it with a few further words in favour of metrics generally. Individual productivity metrics don’t work, but some metrics are still useful, including team metrics which you might class as productivity-related.

• Individual productivity metrics don’t work.
• Productivity-like metrics are still useful…
• …but they don’t tell the whole story

Individual productivity metrics don’t work

Joel O’S states that if you incentivise people by any system, there are always opportunities to game that system. My own experience here is in a previous company where we incentivised developers by how much client-billable time they clocked up. Unfortunately it meant that the developers flatly refused to do any work on our internal systems. We developed internal account codes to deal with that, but it just meant that our incentivisation scheme was broken as a result. Joel has other examples, and Martin Fowler discusses the topic similarly.

Productivity-like metrics are still useful…

Agile development people measure something called “velocity”. It measures the amount of work delivered in an iteration, and as such might be called a measurement of productivity. But there are a couple of crucial differences to measuring things such as lines of code, or function points:

• Velocity is a measurement of the team, not an individual.
• It’s used for future capacity planning, not rewarding past progress.

Velocity can also be used in combination with looking at future deadlines to produce burndown charts and so allow you to make tactical adjustments accordingly. Furthermore, a dip in any of these numbers can highlight that something’s going a bit wrong and some action needs to be taken. But that tells you something about the process, not the individuals.

The kick-off point for Joel’s most recent essay on the subject is a buzzword-ridden (and just clunkily-worded) cold-call e-mail from a consultant:

Our team is conducting a benchmarking effort to gather an outside-in view on development performance metrics and best practice approaches to issues of process and organization from companies involved in a variety of software development (and systems integration).

It’s a trifle unfair to criticise the consultant for looking at performance metrics, but one has to be careful about what they’re going to be used for.

…but they don’t tell the whole story

A confession. We track one particular metric here in the development team at Guardian Unlimited. And a few days ago we recorded our worst ever figure for this metric since we started measuring it. You could say we had our least productive month ever. You could. But were my management peers in GU unhappy? Was there retribution? No, there was not. In fact there was much popping of champagne corks here, because we all understand that progress isn’t measured by single numbers alone. The celebrations were due to the fact that, with great effort from writers, subs, commercial staff, sponsors, strategists and other technologists we had just launched

• a refreshed and reorganised Arts site,
• a brand new Arts blog,
• our first ever dedicated Music site
• with its own Music blog,
• and an exclusive video,
• and we still made time to help our colleagues over at Guardian Weekly launch the new Guardian Abroad site.

A bad month then? Not by a long shot. The numbers do tell us something. They tell me there was a lot of unexpected last-minute running around, and I’ve no doubt we can do things better the next time. It’s something I’ve got to address. But let’s not flog ourselves too much over it — success is about more than just numbers.

Quality seems to be widely misunderstood. People often talk about quality being a key variable that a project team can choose to alter to reshape their project — along with other variables such as cost, scope and time. I think that view is flawed. Meanwhile Mishkin Berteig is not unusual when he says “I have come to believe that quality is not negotiable” — meaning that although it’s possible to negotiate over it, it’s fundamentally wrong to do so. I think that view is flawed, too. From my position quality can rarely vary in the same way as other key project variables, is rarely changed directly, and often it is right to allow it to drop below 100%. I want to spend some time here explaining why.

Before I launch into this, allow me to say that although this is inspired by an issue on which I differ with Mishkin, I do have great respect for him. In fact, it’s arguable that I’m only writing this because I have such respect for him — because it’s so surprising to see him write something that seems to be so mistaken.

The sections ahead are:

• Quality is about much more than bugs
• Quality is rarely addressed at the outset
• Quality is less flexible than scope, time or money
• Quality is usually the driven, not the driver
• Quality can be lowered — and sometimes that’s okay
• Non-executive summary for non-executives

Now, let’s get on with things…

Quality is about much more than bugs

First, let’s make one thing clear: quality is not just bugs. It includes look and feel, ease of use, ease of installation, performance, and so on. Looking at Mishkin’s post it’s clear to me that we he says “quality” he really means “bugs”: he is making a strong case for test-driven development. But test-driven development doesn’t help with an application that requires six clicks rather than one to enter the default data. It doesn’t help that one of your modules needs administrator access to save its state. It doesn’t help that you can’t add any more servers when you need to triple its capacity. These are situations where quality could be improved, but which aren’t solved by test-driven development. Quality is wider than that.

Quality is rarely addressed at the outset

There’s a classic project management mantra in many variations: “cost, quality, time: pick any two” is one such [1, 2 (PDF), 3]. At one level it implies (quite fairly) that the major project variables at the outset are interdependent, and you can’t choose settings for them all. You can’t say, for instance “I want it to do all this, I want it to be flawless, and I want it in a week” — making just two of those demands determines what you get for the third.

But at another level it suggests that people are making these kinds of decisions, and that quality is often determined at the outset. I find this implausible. I rarely hear clear requirements at the start of a project about the quality of the result. I’ve heard vague pronouncements, but rarely anything that could be factored into a project plan. I’ve heard “we need to do better than last time”, and “let’s not over-do it the way we did before”. But that’s usually part of an agenda-setting speech, not the project plan. It’s not enough to help make a calculable impact on scope, cost or time.

At the start of a project you will hear a precise number for a budget, a strong date for a deadline, and (we hope) a clear list of requirements. But it’s very rare that people give a precise level of quality — it doesn’t even have a common unit of measurement (remember: quality is not just a bug count). Quality is rarely negotiated at the outset. The client expects the software to do the job specified, and that’s that.

Quality is less flexible than scope, time or money

Another misconception is that quality can be varied to the same degree as other major (initial, or input) variables such as scope, time and money. Scope can be cut right back, or added to indefinitely. Similarly for money and time. But quality doesn’t vary in the same way. Quality can’t be increased arbitrarily — after a point there’s nothing left to do, assuming the scope isn’t changing. Similarly it can’t be reduced to zero, because below a certain point — well before you get to zero — the product isn’t doing the thing it was supposed to do and therefore you’re reducing scope.

However, while I believe that quality varies less than other project variables, I do believe the consequences of improving quality — even a small amount — are significant. Unit tests reduce firefighting and ensure the project is always moving forward. Usability testing ensures people are much more comfortable and confident with your software, paying dividends throughout the business. Increasing quality by a small amount can have a big, positive knock-on effect.

Quality is usually the driven, not the driver

I’ve talked about scope, time and money being “input” or “initial” variables. They’re things that are usually explicity set at the start of the project. Quality, however, might be set early on, but it’s usually determined by other things.

Sam Newman writes, correctly in my opinion, that quality is driven by process. Here, they way you do things determines how good the result is. Quality is an “output” variable more often than not. Introducing test-first development is a process change; allowing post-development testing time to be reduced because the development phase has overrun is a consequence of an unfortunate project management process; introducing or eliminating performance testing is a process change. In all these cases quality is not an independent variable; it is driven by other things.

Quality can be lowered — and sometimes that’s okay

Quality is negotiable, and it would be foolish if it wasn’t. It would be foolish if budget, scope and time weren’t ever negotiable, but they are, and in that respect quality stands next to them on a level.

Real situation: The website is being built; the investors are making bullish noises in the City; the launch — a release and public demo — was scheduled months ago and is due next week with 200 big media attendees, and testing is still on-going. If a real, but infrequent, bug is found that can’t be fixed by next week does the company cancel the launch?

Real situation: You’ve been working for months on your release. It’s almost done, but there’s another project looming, which the sales team are desperate get started. The time remaining for your current work will leave some untidy edges, but there’s a pressing commercial need to release it and move on to the new project before the company loses more competitive edge than it already has. Do you push for a project extension?

If a developer tells me their professional needs are bigger than those of the organisation in which they work, then they shouldn’t be working there. A developer works for their organisation — just as everyone in that organisation does — not the other way round. And making the right decisions for the business must have positive consequences for the development team.

Mishkin says

Every time the developers sacrifice quality for schedule/cost/features, they incur a debt. Sometimes this is called “technical debt”. I like to call it lying.

Powerful prose, but I’m afraid that particular extract strikes me as uncharacteristically arrogant. “Technical debt” is an honest term. By using this term the developer says to their organisation that the decision taken now will incur an increased cost later. The organisation as a whole can then weigh up the options and make an informed decision. And call me crazy, but it might just be that after being appraised of the situation, the project board, or managers, or directors might be in a better position to make a strategic decision than the developer who’s paid for their expertise in one very particular part of the organisation.

Of course, that’s not to say that every decision on quality needs to be raised to board level. Just as a developer wouldn’t dream of bothering their project board as to whether they should use descriptive variable names (”Well, short names are quicker to type and might shave off 2% of our delivery time, but longer names will save us time if we need to revisit that part of the code…”) so there will always be ways to improve quality in refactoring, or configuration, or packaging, or architecture which the developer should just do without question because they’re a professional. But the developer who never informs their project board of major quality forks in the road, and who never allows their client to be involved in such decisions, is failing to fulfill their responsibility as a member of the whole project team.

At Guardian Unlimited we’re always looking for ways to improve the quality of what we do. But I’ve also heard people say “let’s not gold-plate it”, and “look, we’re not producing art”. These are senior developers who are arguing against particular quality improvements because they are able to weigh up not just the technical decisions but also the wider business decisions. They are being pragmatic. I’d say they are much better developers for it.

Non-executive summary for non-executives

Let’s recap. Quality is really important. It’s about much more than unit testing. And making small increments in quality can result in disproportionately large increments in the results. But let’s not kid ourselves that quality is a variable that can be tweaked arbitrarily and independently this way and that. It’s not of the same standing as budget or scope or time. And let’s not kid ourselves that developers of principle have to stand firm against some kind of management beast constantly trying to undermine them. Developers are an integral part of the organisation, and should behave (and be treated) as such. The quality of developers’ output needs to be balanced just as every other organisational demand does.

As long as we can see and understand software quality for what it really is, the better we are able to improve it.

Previously I wrote:

Sometimes you work with something for so long that you forget that other people aren’t familiar with it. This happened recently talking to a friend of mine — “I’ve got no need for object orientation”, he said, and I was shocked. [...] What, I wondered to myself, were the benefits of OO that he was missing? How could I persuade him that it does have a point? So here is my attempt to trawl my memory and see how I’ve changed in response to OO, and why I think it serves me (and my projects) better. In the end I’ve found that the benefits are real, but nothing I would have guessed at when I first started in this field.

I’ve already written about data hiding, separation of concerns, better testability and refactoring. Now read on for the final part…

Evolving the product

The final advantage to OO: it’s easier to evolve — hell, just to change — what you’re producing. Now ideally this shouldn’t matter. Ideally you’d get your requirements, your client would say “I want to this, this and this” and off you’d go, and then you’d produce something which the client is delighted with first time. That works in a waterfall environment, an environment where things are predictable and don’t change, but not in most cases. In most cases the client changes their mind, or adapts their ideas, and that’s just reality.

Note that I’m not talking about the so-called advantage of OO that I’ve often heard cited, and never really believed: that it’s easier to capture requirements with OO because it better models the world. To me the advantage is in tracking changing requirements.

I think OO deals with this better because people’s thinking evolves around things more than processes, so the amount you have to change your code is roughly proportional to the amount of rethinking your client has done, and their idea of a “small change” will at least be closer to your idea of a small change.

Again, this doesn’t come for free. You need to keep the structure of your application running in parallel with the structure of the client’s thinking. This takes experience and some work. But at least you’ve got more of a fighting chance of success than with a procedural language that’s already one step removed from people’s thinking. In particular domain driven design is all about ensuring the structure of software keeps up with how clients understand their domain, with the express aim of making change much easier. And of course being able to refactor with confidence is an important part of that.

Why wasn’t all that so obvious?

If you’d have asked me five or ten years ago about the benefits of objected oriented programming I could have cited the theory but wouldn’t have really been able to justify it. Data hiding is helpful, but probably doesn’t warrant changing your language — and your thought process — alone. But ideas such as dependency injection, refactoring and domain driven design are things that have come to prominence only relatively recently. They’re where I see the real benefits, and none are obvious from the atomic concepts of OO.

None of this means OO is always the right answer. If your requirements are solid and your domain is well understood then lots of the above advantages are irrelevant. But most of the time that isn’t the case, and it’s then that object orientation can show its value.

It’s odd to me that OO’s benefits are really only tangible years after it’s come to prominence. Perhaps it takes years for us as an industry to really understand our tools. Perhaps it’s only after the masses have taken up a technology — and produced an awful lot of badly written software — that the experts will come forward and really explain how things should be done. Or perhaps I’m just a very slow learner.

Whatever the reason, I can at last explain why I think OO is a step forward from procedural languages. Even though it’s not obvious.

All installments:

• Part I of V, data hiding
• Part II of V, separation of concerns
• Part III of V, better testability
• Part IV of V, refactoring
• Part V of V, evolving the product (this installment)