A quick thanks to the folks at ProgrammableWeb for making my Guardian Tag Bubbles app their Mashup of the Day. It was terrific to see it up there with the likes of the imaginative The Sheep Market and Again But Slower — to pick just two at random.
I was by turns initially horrified and puzzled when I read Ryan Sholin’s piece on “How to manage technology decisions in 5 easy steps”. Horrified because they seemed at odds with my own experiences of what works and what doesn’t, and then puzzled because Ryan is someone with a great deal of experience in digital media, so I couldn’t understand why he was writing these things. Eventually they made sense, but only when I realised his tips were for entrants of the Knight News Challenge (KNC), which is a very specific competition with very specific demands from an audience being mostly of a particular type.
It became apparent to me that much digital media advice for non-technical people really needs to come with a very strong health warning. Here’s one taken from a prescription medicine I’ve just retrieved from my bathroom. It needed only very light editing and seems quite appropriate:
REMEMBER this advice was prescribed only for you. Only a digital professional can prescribe it. NEVER give it to someone else to use even if you think their project is similar.
Looking at Ryan’s blog the advice is for “dealing with developers and choosing a platform for your [Knight News Challenge] project”. As a previous winner of the KNC he should know what he’s talking about here. Unfortunately in the process of being prepared for the KNC blog its intent seems to have been overstated. It is not, unfortunately, advice on “how to manage technology decisions in 5 easy steps” — which is a shame, because I and a lot people I know could do with easy answers to hard technology decisions. Nor does it provide guidance on “How to hire developers”, as the intro suggests, unless you’re hiring one or two developers specifically for the KNC.
This may sound flippant (perhaps, admittedly, because it is) but there is a serious point here. I suspect there are going to be many more people following the KNC than there are participating, and I think a lot of those followers will be people in the process of embracing digital media projects for the very first time, and will be looking for guidance. Perhaps they are just starting on digital projects within their current traditional media companies. They will need good guidance so they can make a reasonable success of their early projects and so feel confident about getting more involved. The media industry needs those people to be successful.
But taking the right advice for the wrong project will lead to problems. Here are some instances of Ryan’s advice and where your (or a colleague’s) situation might require something different…
1. Learn a little bit about any one Web framework, standard, or programming language
If you’re leading a 2 or 3 person team then it’s a good idea to understand the technologies your team members are using. But you may find yourself in a slightly different position in a larger team. In that case the level of conversation you need to have with people will be different. If the technology being used is fairly complex then deep-diving into a particular web framework or programming language is going to be far less useful than having, say, a good overview of the technologies involved and why they’re important — if only so that you can ask appropriate questions and deal with the answers appropriately.
2. Choose the people you want to work with and spend an ample amount of time telling them what you want.
I’d never disagree with the “ample amount of time” part — if you don’t dedicate a lot time to your project don’t expect it to go the way you want. But the “telling” part will only get you so far. It will probably work very well if you’ve become an expert in a particular web framework which is the basis of your project, and if you’ve hired someone with less experience there. But if your situation is different your approach should be different. For anything ambitious or complex or imaginative the development will be an evolving two-way conversation, not a one-way monologue in which you’re telling someone what to do.
So, lots of good advice on the KNC blog for participants of the Knight News Challenge. But if you’re sitting on the sidelines you should not mistake the demands of the Challenge as being the same as the demands of whichever project you happen to be working on next. Get advice that’s been prescribed specifically for you.
This information has been provided free of charge under the National Digital Health Service.
I’ve been reading a bit about lean working recently, and this is a bit about what I’ve learned.
Lean software development is a fascinating step on from Agile, but its history is in manufacturing cars and to date I’ve only been reading up on lean manufacturing, not lean development. There are two reasons for this. First, I’d feel much more confident about lean if I know the background and reasons for things — in other words, I want to derive lean software development from first principles. Second, I was rather cautious, if not downright sceptical, that something can be translated from manufacturing to software. So if there’s any translating to be done then I want to do it myself — or at least understand someone else’s translation of some of the processes sufficiently to trust the rest of their translations.
This, then, is some of what I’ve learnt about lean working so far, based on manufacturing and with my own projections onto software development. It comes with the caveats that it’s partial and not informed by 99.9% of the work Mary and Tom Poppendieck, the acknowledged leaders in the field of lean software development. But it is informed by my reading of two fabulous books: The Toyota Way by Jeffrey K. Liker (TTW), and Lean Thinking by James P. Womack and Daniel T. Jones (LT). My interest in cars registers on no scale visible to the human eye, but learning about car production from these authors has been quite eye-opening.
In this post:
The “common sense” way of putting a complicated thing together is to divide the manufacturing process into pieces (the engine, the chassis, the interior, the shell), create a specialist team for each area with their specialist tools, then have them work in a long production line. The result is a very long production line with each product moving from one specialist team to another as it’s built up from beginning to end. If you’re going to be really efficient then you might batch up all the products of one type and send them through in one go, then switch to another product of another type and send those through.
Lean says this is wrong. It’s not trying to be deliberately challenging or quirky or iconclastic just for the sake of making a name for its founders. Rather, it says this is wrong because it generates hidden problems. If we look at the very long production line then every one of the products on that line is in an incomplete state and hiding a multitude of problems. And if you were to implement that single-product-batching it would be even worse.
Lean says the most efficient way of doing something is to create it from beginning to end in a single pass. The single long production line transforms into many, much smaller, teams that are each responsible for the entirety of the production of each product.
This is counter-intuitive. Did we learn nothing from Henry Ford? Well, the evidence — and there is plenty of it — says otherwise.
But I’m getting ahead of myself. Let’s start at the beginning…
This is pretty unstartling and almost uninspiring. But it turns out that what constitutes value is often hard to define. First you have to ask “value to whom?” Often in large organisations people are set targets of something which is not the main goal of the organisation, and this means you end up optimising your processes for the wrong thing. For example, the paint team might be targeted on how quickly they can paint a car body before moving onto the next one. But that would be misleading, and might explain why there’s a retouch team further up the line to cover up the paint problems discovered later. A car company is not about producing painted bodies, it’s about producing cars, and focusing a team on painting ensures they miss the big picture.
The Guardian software team produces software, so it’s easy to see that’s where our value lies and we should optimise our processes around software. But on second thoughts I’m not entirely sure… Perhaps value should be measured from the point of view of the end user. The better we provide news and information the greater the value of what we do, and merely producing software is no good if it doesn’t benefit the end user. And when we do produce software it’s of absolutely no value until it’s released and public.
Lean is about the value stream
Once you’ve identified the value you need to map the value stream.
Let’s suppose our value is in producing software that the end user finds useful. The value stream runs from the point of idea inception to the point the user can use it. Mapping the value stream is the painful process of tracking exactly what happens to what and who, and how long it takes. This is painful because you need to be very, very honest with yourself, and map with your eyes wide open. In particular you need to make sure you track all the times something goes back for rework, and all the times the thing just sits there waiting for something to happen to it. It’s also painful because it means managers have to understand first hand what their team have to deal with — anyone who thinks they are above that due to demands of time or priorities is going to find this first step very difficult.
A typical Agile process has a backlog of work cued up before the iteration or sprint, and then hopefully delivered at the end. But there will be large gaps between a story being defined and actual development. If an iteration is three weeks then on average there will be a gap of 1.5 weeks between definition and development starting, and if we insist on any story being up to five days’ effort then a typical optimum scenario is that a single story is at least 60% non-work. This is relevant because…
Lean is all about eliminating waste
The primary purpose of lean is to eliminate waste, and I’ve not put it first in the list of things I know because we first need to define value and then map the value stream. Once we’ve done that we can tackle the waste.
Waste comes in many forms — well, seven forms according to lean lore. In manufacturing the most significant of the seven is unused inventory, which means parts that are sitting around in racks and warehouses waiting to be used. Unused inventory not only takes up unnecessary space, it actually masks problems. Here are some examples:
Inventory in manufacturing seems similar to Agile stories that aren’t being worked on: they’ve been specified but are sitting around waiting to be picked off the warehouse rack. And when they’ve been developed they might be sitting around waiting to be released — after all, until it’s released it’s of no value to the customer.
This is where we return to the example above of a story that’s 60% non-work — 60% of its time is just sitting around waiting. The goal is to compress this down to an ideal 0%, but not just because we want to do things quicker. It’s because having sight of something from beginning to end, and not lose sight of it for a second. It’s because being able to focus on something means information isn’t lost, and everyone’s expertise can be brought to bear on it in one pass. If that were to happen then less would be needed to be written down on the story card because the team wouldn’t have to suffer context switching. They could also apply much greater creativity to their work, because they would see exactly why certain things were and were not being specified and contribute alternative or additional ideas without causing confusion or risking repetition.
This idea of taking something from beginning to end in a single pass is called…
In manufacturing this is a big deal. It means rather than having your factory floor as a single production line you create small teams (”cells”) which are responsible for the entire production of each unit (car, lawnmower, etc). If you have huge lathes and paint machines and so on it’s a major change to rearrange the factory floor.
Not so difficult in software development, fortunately — we tend to just have desks and computers, though in any large organisation with centralised functions you always need to win the buy-in of other people.
However, flow comes with a new and serious responsibility for those involved: the cell as a whole is responsible for producing the goods, so they must work together to ensure regular and maximal output. Let me make that concrete…
At no time when I’ve been working with cross-functional teams (software developer, client-side developer, QA, etc) has there been the perfect balance of all roles; we could always have done with one more software developer, or an extra 0.2 of a QA, etc. Much of the time the imbalance is negligible (or quietly welcome), but sometimes it’s very noticeable. And when a cell is working together on a single deliverable (the car, the lawnmower, the software feature) then it’s up to everyone to help each other. It’s no good the client-side developer producing more widgets to test if the QA can’t keep up. They need to ignore the traditional perceived boundaries created by job titles, reach across to others, and work together to regulate the output.
I said lean is about eliminating waste, and flow helps with that in a way hinted at earlier. Flow increases quality by allowing all participants to see the thing put together from being to end without interruption. This reduces hand-off time, reduces information loss, reduces relearning, and increases knowledge and ownership. The number of times an item needs to go back for rework is reduced, and if it does need to go back then the rework that’s needed is clear and therefore quicker.
Meanwhile, all this work needs to come from somewhere and that’s what “pull” is all about. The principle of lean is to only do what’s needed, and that means only produce something that is a direct response of a specific request, and only when needed.
The distinction is clear in car manufacturing. A car company’s marketing department will devise a special offer on a particular configuration (these seats, those mirrors, any one of the following colour combinations) and the plant will have to manufacture a whole lot of those particular cars ahead of time, but only with a guess (rather than a certainty) about what the demand will be. In the lean world a car is only produced in response to a specific customer’s specific purchase: customer goes into showroom, customer orders car, order triggers build.
That’s how pull works in relation to delivering the product to the customer. But pull also works in relation to building the product inside the factory. The old method is to keep hundreds of each part in store; the lean method is that a part is only provided to the worker when they need it: when they’re running low they signal the need for more, and it’s provided for them. This triggers a chain right back potentially to the supplier of the part, ensuring they are always delivering just enough, no more and no less.
The parallel flawed system in the software world is the product backlog. (We’ll ignore the even worse scenario of waterfall’s detailed planning up front.) Work is prioritised ahead of the sprint and waits to be developed. Requirements can change, even in that gap between prioritisation and development. The consequences aren’t as terrible in the Agile world as in the waterfall world, but it still causes problems: it disrupts the team’s schedule and of course all the effort that went into the planning of the now-deprioritised story is wasted. Even if the requirements remain constant the gap between planning and developing mean knowledge is lost, or conversations need to be repeated, or the requirements turn into mini-waterfall-style requirements specifications.
The lean software alternative is to prioritise the next story only when the team is ready to work on the next story. That means while the team is developing the current feature they don’t have much certainty about what’s coming next. Like the worker in the car plant they have to signal slightly ahead of time that a new story needs to be worked out. Then the Scrum Master/business owner/internal customer needs to get something ready so that when they become available the team can all get together, thrash out the details, and set to work again.
As a manager I’m uncomfortable with this: I can no longer know what’s going into an iteration when the iteration starts. But the definition of value is not “what a manager’s comfortable with”. I will need to find other ways to ensure we can be accountable to the rest of the business, and they will need to be ways which are closely aligned with end user value — and that can only be a good thing.
The combination of flow and pull doesn’t mean the team is only working on one thing at a time. But it does mean that everyone has an equal balance of work at all times. So if Alf’s development work gets passed over to Betty’s testing then Alf and Betty need to make sure that she is expecting to finish her current piece of testing at pretty much the same time as Alf finishes his current piece of development and passes it over to her. Keeping that flow even is really important.
Lean promises the seemingly-impossible
Lean holds out the seemingly-impossible promise of increased productivity and increased quality. But here are some numbers from the literature:
I do have some concerns about lean software development, but they’re less about lean itself and more about bandwagon-jumping and doing things without really understanding the reasons. Regardless of that, it’s refreshing to find a new way of looking at what seem to be known problems, and making insights which you might not otherwise have found. It’s certainly something I’ll be spending quite a bit more time on.
