Here’s the test question: You run a company that creates software in high level languages. You have a collection of developers who want to work for you. How do you sort the ones who are most suitable from the ones who are least suitable?

The answer is to query each one in turn using questions that most closely match the actual work they’ll be doing.

If you need them to design domain models then give them a toy domain and ask them to model it. If they need to work particularly closely with other developers then get them coding next to one of your current team. And if, for some perverse reason, company policy bans the use of software libraries and they have to implement every basic software function from scratch every time, then it’s a good idea to ask them to write a sorting algorithm.

But if that last situation is not true of your company, then the correct answer to “Show me how to sort an array x of integers” is “x.sort()”. Or some variant according to your local tongue.

If you get that question wrong, then unfortunately you fail, and miss out on some really excellent hires.

One major problem with targets comes about when they are different from the core thing the company is doing. For example, the number of features delivered by a software team may be a rough guide to their productivity, but is the team’s productivity really helping the customers get their widgets? As soon as you target the team members on this figure they will aim towards it to the exclusion of the end goal. You also end up creating or reinforcing a silo — suddenly the team is no longer interested in helping customers get widgets, it’s a team whose purpose is productivity, divorced from anything else.

A key phrase to watch out for is “is a proxy for”. For example, in a call centre: “Call duration is a proxy for resolved problems”. As soon as people are motivated towards a proxy the real target is forgotten.

At Øredev 2010 John Seddon gave an example of a service team whose managers thought they were meeting 100% of their targets, closing all their cases in 25 days or less. But the real work was not about closing cases; it was about resolving their customers’ problems. When they looked at the real activity that went on, they found that a single customer’s problem might be broken into four or more cases precisely because the staff were motivated by a 25 day case-closure time. From the customers’ point of view their problems were really taking 50 or 100 or 150 days to resolve. False targets obscured the reality and let an failing system remain in place. John says (at the 33:34 point):

If you hold people to account with arbitrary measures they learn to do anything to meet the arbitrary measures, and that is not the same as serving the customer.

There are other problems with performance targets, but that’s enough to be going on with.

When implementing a change of working a while back in my development team my boss of the time said, “Well, okay, but I want you to show me that your changes are making a difference”. What’s the metric for better software? I knew all about the dangers of measuring things like lines of code — that wouldn’t do anyone any good.

So I opted for something low-tech. I asked people. I asked team members and wider stakeholders once fortnight: “How are we doing?” I broke it down into ten questions, varying depending on the role of person and over time I could chart our progress. We went from around 30% satisfaction to 80% in four months and then plateaued. The plateau gnawed away at me, but by that time interest had moved on — the team was doing a great job and there were more pressing problems elsewhere.

I subsequently spoke to a programme manager who used a similar, but simpler, technique. He asked: “How would you score us out of five this month?” And then the follow-up question, if it wasn’t full marks… “What would we need to do to make it five?”

I much prefer this version, which is simpler, more direct, and there’s more clarity on what to do about the results. It reminds me of the guerilla approach to lean product development: putting a mock-up in front of someone in Starbucks and (eventually) asking “Would you buy this product? What would it take for you to buy it?”

I’ve used lots of other metrics for different things relating to performance, output and productivity: bugs resolved per week; lines of code per class, and so on. But I don’t think I’ve ever found any for this kind of thing which are so simple and connect the work and the output so effectively.

I was thinking about this after reading Mark Foden’s excellent response to Sir Gus O’Donnell’s recent Telegraph article. If you’ve not read it, you should (Mark’s piece, but also the former Cabinet Secretary’s). The structure of Mark’s response is: (1) Sir Gus O’Donnell is another voice saying government needs to take more risks; (2) the truth is government needs to take fewer risks, which it can do through (3) trying things out in a small way and then learning and adapting, over and over again.

In fact this is a fascinating example of “what they say, and what we hear”. Here is a key part of Mark’s setup:

I infer a model of thinking that goes: The old ideas aren’t working any more… so we need to be innovative and adopt radical new ones… because the ideas are new we don’t know what will happen so this is risky… but we have no choice so that’s OK… if things go wrong we must be mature and learn from the failure rather than throw stones… next time we will know better.

When I looked for those words in the Telegraph article I couldn’t find them. And then I realised why: I was caught out by “I infer…”. Sir Gus didn’t say those words, but Mark heard them (or, more correctly, heard an echo of them).

Similarly there are many times when I’ve heard CEOs and others saying “we must take more risks”. And in retrospect, I think they didn’t actually say that — at least not in the way most of us understand it.

When we hear business leaders talking about taking more risks what they are actually saying is: we must move out of our comfort zone. More broadly they are saying: we have become stuck in our old ways, and become too comfortable, while the world around us has changed; the old certainties are no longer with us so we must work without the old systems that grew out of those old certainties.

And of course, working without old certainties feels like taking more risks. Indeed, less certainty is one key part of risk. (The other is undesirable consequences.)

If you gave pressured CEOs a hard choice between increased risks and a change in the way things get done, I think most of them would opt for the latter.

So many organisations are being disrupted — by digital startups, the internet generally, the economic downturn… The civil service is just one of these. Most leaders facing these problems do not want to increase the risk profile of their organisation. What they want is to sweep away their organisation’s old systems and embrace new ones. That might seem risky to some, but it doesn’t need to be.

To use an analogy: If you meet someone who will only ever cross roads at a pelican crossing then you might see them as someone who is stuck using old, time-consuming methods. They might see you a maverick risk-taker. But if you teach them the Green Cross Code they may learn that there are other ways of achieving their ends without taking undue risks.

Gus O’Donnell did indeed use the phrase “take more risks”, but he used it in the context of breaking away from cumbersome systems of red tape, and so I think he meant it more in the sense of moving out of a comfort zone. He also referenced one of the two keys to success that Mark mentioned: learning from experience. So I don’t think he and Mark are very far apart — if indeed they differ on anything much.

Similarly staff who hear their bosses say “take more risks” might want to reconsider what’s actually being said. Usually it will be: let’s do things differently.

Having often listened to John Seddon I’m wary of “lean”. He, too, learned from Ohno, and regards lean (which is the production approach extracted from Toyota) with disdain. His view is that lean is a version of Ohno’s approach which has been packaged by business school professors to sell to Western executives who want easy answers handed to them on a plate. Ohno’s real approach, he says, is to get people as close to the work as possible, and let them solve any problems that arise.

So while reading the summary of The Toyota Mindset I was curious as to what I would find, especially as it proclaims to hold “The Ten Commandments of Taiichi Ohno”. That sounds a bit packaged-for-resale to me. But in fact I was pleased to be wrong.

The “ten commandments” are much more ways of thinking, and ways of approaching business problems, than actual commandments. Two things really stand out from the stories of Ohno that Wakamatsu relates, via Abilla.

First, the number of times the phrase “the shop floor” appears. This is the reason lean might be considered packaged for resale: because Ohno’s real lessons are about getting onto the shop floor and seeing problems first hand, something that most executives in large companies would like to think they are beyond. Some examples from Abilla’s 10-part review:

• On standard work: “Standard work must be realistic and applicable on the shop floor”
• On how to know things: “Taiichi Ohno believed that one should base their judgments on his or her experience on the shop floor, not from a document alone.”
• On learning from the masters: “What you read from books is not usually useful when it comes to improving the shop floor. You will find much better ideas by just trying different methods on the shop floor.”
• On truth and understanding: “Stand and Observe the Shop Floor”
• On disclosing mistakes: “One day, Ohno stepped into the shop floor…”

Okay, that last one wasn’t such a big deal, but it’s notable that there aren’t any Taiichi Ohno stories that begin “One day, Ohno was sitting in his office…”

The second thing that stood out for me was how frustrating it must have been to work for Ohno. There are so many stories in which he asks an employee to do something, they do it, and he scolds them for doing exactly what he says (and in one case, for doing it immediately).

But the lesson here is that the staff should be thinking for themselves, solving problems for themselves, and always going beyond mere instructions.

Of course there is much more to the lean — sorry, the Ohno — way of thinking than these two observations. There is, for instance, the mindset of the continuously watching for waste and acting on it.

The stories related by Wakamatsu seem to be fascinating, and it’s yet another book to add to my growing reading list.

• Space to learn. Startup veterans are always quick to say how much they’ve learned — often from their failures. We need to give our teams space to learn, too. Only then can they really own the knowledge and take it beyond what we intended.
• Communication to the team. To ensure the team has your vision and can feel part of the long term picture while they deal with the immediate issues, continual communication is key. Gossip adores a vacuum. Also essential here is honesty. Honesty simplifies explanations when things go awry later, and forces us to clarify our own thinking and confront your own prejudices.
• Let the team present. Knowledge sharing is important, and that’s forced out when this happens. Encouraging the team to present externally also shows them independent validation of their expertise.
• Proximity to the product. Technology is great, but technology has to be about something. By keeping the team close to the end product and the end users they become much closer to the business, and therefore much more valuable. It also provides many more opportunities for innovation.
• Job titles. Less about creating brilliant teams, more about maintaining them. It’s important to understand that the demands on people will change, particularly in a small company. So make sure job titles are future-proofed, and you’re not going to have to give someone a lesser job title when things do change. If you’re a three person startup and the most technical person is the only developer, then should they be given the title CTO? It might help secure them, and it fits their current status, but is the greatest developer the Chief Technology Officer of Year 2 or 3?
• Remind them they’re brilliant. It’s easy for the team to get lost in the details, and the pride they have isn’t built on the same things that your pride is built on. So take time to remind the team that they are brilliant, and show the team why.

There were lots of smart questions, too.

You can also see great presentations from…

• Andrew Nesbit, of Forward Technology: Form Analytics With iForm
• Avi Patchava, of LinkOut: Online Connections to Offline Relationships
• Azmat, of CityMapper: How to get to #1 in the App Store

Finally, we were entertained by Shed Simove, who talked about his life of pushing boundaries in the name of having a lot of fun (mostly). If you have the opportunity to see Shed talk, then go.

When referring to Doug Hubbard’s book “How to measure anything” I sometimes uncharitably say he performs some sleight of hand right at the start, part of which is to insist that we must be measuring with the intent of influencing some concrete decision. It’s sometimes difficult to explain why that is so important, but Paul provides good examples.

He describes his conversation with “a doughty advocate for public transparency”:

Our debate arose from his astonishment that it wasn’t possible for “government” to say at any one time how many people it employed. Despite this being an “obvious” factual issue in his eyes, no amount of requests seemed to be able to produce a meaningful answer.

My response “well, it’s not really a meaningful question” – didn’t go down too well.

Paul lists a number of reasons why the word “employed” is ambiguous “with all its colour and texture of vacant posts, secondments, part-funded posts, long-term absentees and part-timers”. The word government is similarly problematic (as indicated by Paul’s use of quote marks above): does it include people employed by local authorities? Quangos? Working in NHS hospitals? In government-funded voluntary organisations?

It was this segment that caught my eye and triggered my post here:

If asked by an economist with a specialism in operational research or organisational productivity, I could possibly, possibly see some sort of tangible purpose to a question, but more likely a version targeted at a more specific organisation or sector than just “all of government”. Possibly.

Indeed: possibly. If that economist was on a mission to do something specific. Then we could nail down the exact definition of “employed” and for “government” depending on what concrete thing they wanted to achieve.

Here are some things someone might want to achieve, and what we therefore might (arguably) include or exclude in our “employee” count.

• We want expand the public sector because we think that will boost the economy. Include the number of people directly on the payroll of local and national government, and all services funded by local and national government. Pro rata for part-funded services. Exclude services contracted out. We will then seek to increase this number.
• We want to improve public service by putting more staff on the front line. Add up the number of people who deal directly with the public as a key part of their job description (which is a questionable definition in itself). Include people whether or not they are directly employed. We will then seek to increase this number.
• We think big government is bad, so we want to check we’re spending less on people. Add up all costs spent directly on people. Include contractors employed directly. Exclude people employed by companies we are contracting out services to. Include or exclude local government depending on whether “big government” includes local government. We will then seek to reduce this number.
• We want to decentralise decision-making, so want fewer people employed centrally. Count the FTEs performing functions at a national government level, regardless of whether or not they are directly employed. Exclude anyone not working at a national level, such as those in hospitals and local government. We will then seek to reduce this number.
• We want to write a Daily Mail article with maximum harrumphing. Count the number of people working on any kind of government-backed projected. Include people regardless of whether they are employed directly or work for a company contracted to fulfill a service. Count everyone equally, whether they work full time or one hour a year.

I have a number of concerns even with this list, not least of which is that I don’t think any of the proposals is complete in any way. But it does show that the specific, concrete purpose does influence exactly what you measure, and conversely that having no such purpose makes it impossible — certainly in this case.

The purpose of Paul’s article, by the way, was really to say that a quick and easy answer to almost any question about government technology is a near-impossibility, and that this needs to be explained in ways that resonate with harrumphing Daily Mail readers. Addressing that is not the point of my article, but perhaps the examples above are also the start of how we might achieve that.

To review: Netflix want to shift their business to video streaming, and away from DVD delivery. As part of this they’re reducing the service to DVD customers so that you can no longer add to your wishlist (aka queue) from devices which also allow streaming. There is much wailing and gnashing of teeth.

What’s remarkable about this is that Netflix is only young (14 years old) and yet it’s having problems transitioning from a now-legacy position to something more current. The irony is that Netflix itself was once the radical disruptor and dramatically changed the film rental game. It would like to change the game again, but it’s not so easy this time around.

The darlings of the tech media are the young startups — what innovation! What promise! And the cutest stage-school darlings are the lean startups — thrill as they pivot when faced with insurmountable problems!

But these media darlings too often lack a certain… revenue stream. Or reliable customer base (which is what Netflix has). Or any other success factor that’s generally considered meaningful. And as soon as any business hits on that then they’ve got something they need to protect. And that makes change really difficult. All that promise and innovation and game-changing energy is suddenly anchored by the need to retain that success factor — that thing they’ve worked so hard for for so long.

That’s why any successful business is a legacy business.

There is one thing which will prevent this, and that’s if the business is explicitly designed to continually change. Otherwise even very young companies will find they’re facing similar problems to very old ones.

Contents

• Definition
• Purpose
• Varying communication
• Traceability
• Success factors are relative
• Leaders are responsible for balance
• Application to the financial year

Definition

Critical success factors are those factors on which a project’s success is judged. These may include reducing a costbase, increasing efficiency, opening up a new market, etc. Knowing what they are, and putting them to work, become especially important in complex projects where there is the potential for so much noise and error.

Purpose

Critical successful factors can be used for two things: governance (ensuring the programme develops in the right direction) and communication (keeping everyone focused on the right thing).

Of course, there is an interplay between the two. Using CSFs as a framework for communicating to the steering group ensures better leadership (governance). Using CSFs as a framework for communicating to the operational team means better decision-making (prioritisation) on a day-to-day basis.

Varying communication

How CSFs are then communicated will vary, dependent on the audience. For the governance group it can be part of a formal process. For the operational team it might be cards stuck on a wall.

During the Guardian’s R2 programme we stuck the CSFs on a wall, ensuring everyone who cared to look knew exactly what our priorities were. At the time we called them key performance indicators (KPIs), although I think that was wrong: KPIs are usually regarded as continuously measurable healthchecks. I think “critical success factors” would have been a better name.

Traceability

Communicating a project’s critical success factors helps traceability, which I interpret as being able to trace the line from what I am doing now to what this programme is all about. However, it was pointed out that traceability does not appear in traditional agile literature. I think that’s because agile literature tends to start with software development and peters out as you go up the management stack. It’s a failure of the literature, and a reminder that no methodology at any level should exclude good management and leadership.

Success factors are relative

Any individual’s personal success factors are going to be based on their work and influence. A project manager will be concerned about budget and deadlines even though those factors may be secondary (non-critical) to delivering a particular capability or enhancing brand reputation may be more important. The project manager is focused on budget and deadlines because they feel that all they have the authority to influence. That’s clearly not ideal.

For the R2 programme removing legacy systems was an acknowledged success factor. Unfortunately for many technical people it was also acknowledged not to be critical. This certainly influenced prioritisation, and we ended up not decommissioning as much as we’d have liked. A shame for the techies, but we knew we had achieved success overall.

Leaders are responsible for balance

Given that individuals have their own success factors, it’s a leadership responsibility to ensure the right balance is achieved, particularly when there are obvious conflicts. A team lead must balance between feature delivery and test time. A project manager must balance between supplier responsibility and internal effort. A programme manager must balance between budget pressure and ease of business change. The governance group must balance between stated objectives and new opportunities.

Throughout the work there will be conflicts of objective, and the entire project depends on those individuals sitting above the level of the conflict to make a decision and keep it going in the right direction. This is a responsibility of leadership.

Application to the financial year

We’ve been focused mainly on critical success factors for projects: time-limited activities with ringfenced budget. But critical success factors can equally apply to a company’s financial year. If you’re budgeting for something shouldn’t you be defining success criteria for it? And if you’re budgeting for a 12 month period, shouldn’t you be defining success criteria for that 12 month period? If not it’s a gift, not an investment.

So, a lot to think about. But perhaps one thing stands out among all these: actually knowing what your success factors are is the essential first step.

1. Adequate technology hobbles the business;
2. A strong technological backbone opens up new opportunities;
3. Technological short-termism is really costly;
4. The business strategy needs to include the CTO;
5. All this is especially true of non-technology companies.

Setting the scene

Here’s how the WSJ kicks off the story:

As eBay Inc. prepares for a critical holiday shopping season, the company this week plans to unveil new elements of an overhaul in how shoppers find and buy products on its Web site.

Behind the new look, which includes eBay’s first major home-page redesign in nearly four years, is an urgent effort to close a technology gap that has caused the onetime Web pioneer to lag behind rivals like Amazon.com Inc. Amazon for years has had many of the same features that eBay is adding.

Not only is eBay playing catch-up, this kind of work is incredibly disruptive internally:

To change a website’s underlying technology is “one of the most difficult and dreaded things you can do” as an information-technology manager, says Rob Enderle, of tech consulting firm Enderle Group. [...] “It’s like having the jet engines changed while the plane is flying,” [eBay technology chief Mark] Carges says.

They’re dealing with it, but still, it’s not a great position to be in. What can we learn from this…?

Lesson 1: Adequate technology hobbles the business

It’s not as if eBay’s technology is poor. After all, how many people can claim to have a system which handles  “hundreds of millions of users worldwide, over two billion page views a day, and petabytes of data”? Yet while it might have done the job, it clearly didn’t do anything more than that:

EBay’s system, which involved 25 million lines of inflexible code, soon became a liability. The company, for example, couldn’t figure out which of its hundreds of thousands of “iPod” listings were for a given model or for iPod accessories.

The original system was developed only with a the immediate business goal in mind, and each time a feature is implemented with only a short-term view it shuts another small door on a potential new development. Eventually you end up with an inflexible system and the only way to explore a new direction is a with a big, disruptive project.

The system has been continually developed, of course. It has grown to handle a huge userbase and it’s had many enhancements added along the way. But fundamentally it’s always has been developed to do the original job and little more. And as result all eBay, the business, could do was… more of the same.

Lesson 2: A strong technological backbone opens up new opportunities

While merely-adequate systems narrowed the company’s future, could it really have been any different? You can already find products on eBay, so why is it “building a product catalog for the millions of products sold on eBay”? What’s the big difference between finding an iPod in an auction, and finding all iPods of that model? Well, while eBay was focused on bringing each buyer to an individual seller

retailers like Amazon focused on new, fixed-price merchandise, relying on organized catalogs that let the online retailer keep track of what it was selling. That created the ability to cross-sell products and recommend other merchandise.

The big opportunity lost is the ability to cross-sell. If you can’t aggregate similar iPod models a whole avenue of customer insight is lost to you.

What’s interesting to me about this is that the business opportunity arises from strong information architecture within the product database, and pretty much no-one in the company other than a technologist is going to give more than two seconds thought to that kind of thing. Enabling the technologists to think beyond the immediate, and allowing them to build on this, might have meant eBay’s problem would have been avoided.

Lesson 3: Technological short-termism is really costly

Correcting this is more than a big project for eBay:

[CTO Mark Carges] ended up hiring 150 engineers. Mr. Carges also had eBay purchase Positronic Inc., which has built predictive models for financial-services clients.

I’d say that was pretty expensive.

Lesson 4: The business strategy needs to include the CTO

We can be pretty sure the senior team at eBay didn’t just wake up one morning and notice Amazon was leaping ahead of them. They’ll have worried about that for a very long time — Amazon have been cross-selling imaginatively since they started. So why weren’t eBay making these changes years ago…?

The effort has had to overcome cultural stumbling blocks. For years, tech staff didn’t attend key strategy meetings, including those in which eBay decided to emphasize fixed-priced goods.

Mr. Carges, the technology chief, is now a key voice within the company and encourages engineers to take the initiative in testing new ideas and adding features.

A strategy defined, which couldn’t be fully executed because one of the key players was missing from the conversation, and nobody noticed for years. It seems so obvious in hindsight, but it’s not when you’re there. eBay has always been run by really smart people — none of us are immune.

Lesson 5: All this is especially true of non-technology companies

Don’t mistake eBay for a tech company. It might do some amazing things with technology (check those scalability stats again) but it’s fundamentally not a technology company. If it were, then there’s no question that the tech staff would have been involved in all the key strategy meetings.

So these lessons cannot be dismissed. It can happen to any of us.

And any of us can learn the lessons:

“One of the most important things we had to do was become more of a technology-driven company,” says eBay Chief Executive John Donahoe.

Even with this major piece of work going on eBay is still in the same business it was before. It’s still not a technology company. But it is more technology-driven.

Let’s hear it for eBay

Finally, a tip of the hat to eBay for sharing their experiences. They’ll come out of this a better company, and everyone benefits from learning from them.