There was a fair bit of criticism last night of the UK government’s approach to agile development, and in particular its use of agile on the huge Universal Credit (UC) … Continue reading Even government beasts benefit from experience
There’s a fascinating piece by Michael Feathers in which he questions the what “inventory” is in the context of lean software development, and concludes that companies should remove the code … Continue reading Don’t confuse waste with inventory
In the previous post I picked up on Aaron Levie’s analysis of complicated organisations and suggested the root cause was departments working to different agendas, compounded by technology which solidified current complicatedness. In this post I want to suggest how these things might be tackled.
First a disclaimer: I’m really not pretending this is easy. Change is difficult, and change on a large scale is really difficult. So these suggestions are really only small tools in any very large (business process re)engineering project…
1. Focus on business change programmes, not IT projects
Previously I wrote about different departments working to their own agendas which causes inefficiencies and complications. Some readers will recognise this as the premise behind systems thinking, which in this context is the perspective that employees’ behaviour is most significantly influenced by the system in which they operate, including their perceived purpose, targets, and so on. The solution, then, is to re-examine not just how a single department work, but how the entire system operates and focus that around the goal.
I’ll avoid the bigger management consultancy territory and instead stick to technology, because (a) this is a technology blog, and (b) I’ve just set technology up as Lucifer-incarnate, entrenching complicated systems. We technologists must bear responsibility for fixing the problem, and the solution is to examine the system as a whole.
Specifically we should stop delivering “IT projects” and start delivering business change programmes. To me the difference is this: an IT project delivers technology into an existing operation, while a business change programme actually questions and improves the way the business works, independent of departmental boundaries.
Indeed a really successful business change programme should break down boundaries, eliminate bureaucracy, and rewire the operation to focus on the real business goal. Layers of unnecessary interface-translation are removed as real, valuable communication takes place directly.
I can think of one project close to me that’s a good example of this. It involves a lot of complex technology, and the people leading it on a day-to-day basis have a thorough understanding of that technology. But it’s not an IT project. The real change is in non-technology teams, who are having to reconsider how they work and what they regard as important — now much more recognisable as the purpose of the business.
Technologists are in an unusual position in that we operate across many departments. We need to exploit that to join people up and make things slicker.
2. Put the flexibility in the people, not the technology
That first tactic addressed the first problem, of departmental silos. This one addresses the second problem, of technology that entrenches complicatedness.
The approach here is to reduce the technology to its core purpose, removing the process-specific operations as much as possible and allowing people to do those. People are much, much more flexible than software (yes, software is soft, but people are softer), so when things need to evolve there’s no dependency on technology changes. The way a business works is constantly changing, so it should be easier for people to evolve than people+technology.
The software becomes Brian Prentice, of Gartner’s, conception of simple software: “when the next additional feature has no particular value to the majority of its intended audience”. The interesting stuff — the complexity — moves out of the software and into the people. I would hope this becomes empowering, rather than mechanical, work, and the people feel closer to the real goals of the business and continually thinking about improving it. Certainly, whenever I’ve seen complex processes taken out of technology and into more physical forms — people using index cards to manage requirements, or whiteboards to track progress on something — I also see people at liberty to be continually creative with their working practices.
That discussion of simple technology takes us right back to Aaron’s original article. There was enough in that original piece to trigger a year’s worth of blog posts, but for me, most of all, it’s important to understand where organisational complicatedness comes from and how it might be tackled. We technologists might be guilty of creating some of the problems, but we can also lead with the solutions.
Aaron Levie, founder and CEO of Box.net, advocates simple technology solutions for large companies. That should be obvious, so why do companies go for complicated solutions? He argues that it’s … Continue reading The complicated organisation: part 1, explanation
Freelance Unbound has some great laws, clearly learned the hard way, about… well, supposedly about website launches with new content management systems. But actually they can be generalised to be laws of almost any kind of big project with a strong technology element. Laws of physics? Laws of the land? Either way, you don’t need to take them lying down. Here are some suggestions about how to fight back…
The point here is that you can’t get people’s full attention unless it’s actually live. That’s virtually a law of physics. The way to fight this is to deliver early and often: start making it real for people as soon as possible, get something out there quickly so they can use it and feel what it’s like, and then build up.
2. It’s nobody’s fault but yours
This is a law of the land, and you can get a change in legislation. If people work in silos then it’ll be easy to finger-point when things go wrong. If people are brought together from across the organisation and are jointly responsible for the project being a success, then there’s a much better chance of success happening. This is because there is much greater cohesion between people with different expertise, which means more fluid communication and much less loss of understanding. So problems are less likely. And when they do happen then you can expect a much more productive reaction from all involved.
3. No-one has thought about [some critical function]
Again, this is a law that can be changed: find out early who all the stakeholders are, and make sure they’re involved. This is a pain, because it’s much easier to get a project going when there are fewer dependencies or teams involved. But inconveniently projects are judged a success at the end, not at the start. So it’s worth foregoing a bit of pain at start — socialising the ideas, seeing who needs to be involved, involving them, listening to them, changing your thinking accordingly — to avoid a lot of pain at the end.
4. [Team which does unglamorous but critical work] is always the last to know
Yes, it may be easier to involve some people only at arm’s length, but the inconvenience you’re avoiding early on will have to be repaid with interest. As before, discovering and involving all the stakeholders early is key.
5. Things you want to change are always hard-coded — and vice versa
You can’t please all of the people all of the time. But you can choose a small number of people and make the system flexible for them. And then you can extend your system a bit so that it pleases a few more people. And then you can continue to extend and expand, including more and more people, and adding more and more flexibility in just the right places to please them, until… well, hopefully everyone agrees it’s okay to stop.
For this approach to work you do have to make sure you have the capability to adapt your system. If you’ve chosen to buy then you need to know it’s sufficiently flexible for your needs, and where it doesn’t flex (or you’ve chosen not to flex it) then you need to make sure you communicate the limitations to all the relevant people. If you’ve chosen to build then you need to make sure you have the skills to build effectively.
6. Your system will always be out of date
Every project is a response to the wider business environment. And the business environment will never stop changing. So the question to ask is: If this project is a response to business change, and the business environment is changing constantly, what resources do we need to continue to put into this system after the project is complete to ensure it continues to keep pace with the moving target it was aimed at in the first place?
I’d say almost any answer is acceptable, including “We’re happy for it start falling behind the moment it goes live, because that bit of our business isn’t the most important part”. That particular answer will probably upset some staff, but the main thing is it’s a conscious decision. What will lead to problems is not answering the question at all.
Mind you, it’s also possible to get that answer wrong. Or, more likely, to incorrectly predict the pace of business change. If that’s a possibility then acknowledging the risk should help secure additional resources in the future.
Agile comes with tons of literature on how to organise work at a very detailed level. How much of this treadmill is activity for activity’s sake, rituals, and religious manifestations of an ‘Agile Sub-Culture’ aimed at integrating a growing workforce? Can this relentless ‘heart beat’ and this esoteric jargon stifle innovation and alienate the very people Agile purports to help? Could more slack benefit a discipline which ultimately relies on people’s intelligence and creativity?
That’s how we set the scene a few days ago when I led a discussion among some peers from different companies entitled “The Agile treadmill: Command and control in disguise?” It was organised by IndigoBlue as part of their Second Wednesday series, under the Chatham House Rule. This is summary of what we discussed, with fictionalised initials for each participant.
In this article
- Origins. How the question arose.
- How useful is Agile process? It’s not all bad.
- Religious Agile, variable results. Examples to demonstrate that religious Agile is somewhat independent of project success.
- So what is Agile, anyway? To be best understood, agile working should be seen not from the development point of view.
- Seeing the vision, from top to bottom. The importance of all stakeholders seeing the whole picture.
- What did we learn? That religious Agile is independent of project success, that agile is a business methodology, and that the vision must be shared.
When people talk about how Agile they are I find this usually coincides with how rigorous their processes are. Some of the largest, most successful Agile projects I know have an exhausting hothouse atmosphere, where everyone’s role is tightly defined and cannot be deviated from.
Have we created a kind of Animal Farm, where a system designed to set us free has actually enslaved us? Does it make a difference if Agile has been introduced by developers or management? Is effective Agile synonymous with a hothouse environment?
Between us we had an awful lot of experience to help answer these questions.
There was a general agreement with TC that there is some real value in having standard processes and practices:
- Rewriting the rulebook every time you start a project or programme is wasteful if you can capture the processes and practices that are known to work;
- It’s easier to bring in external resources if well-known language and practices are being used in the team.
But practices can be unnecessary, and there was recognition that many practitioners have a near-religious attachment to them, and to particular ways of working. As people managing programmes of work we often don’t have the discipline to recognise what the right level of discipline is… and isn’t.
There was also concern that Agile created an alienating language. NA indicated his particular discomfort over the word “customer” because that was often not how people saw themselves or their relationship with the tech teams.
So no-one was anti-process (nor anti-Agile process), and we know it has produced good results. Our own major rebuild and redesign at the Guardian is one example of that, with a very highly regimented process and an outcome that at least met the aims and expectations. There are plenty of others.
On the other hand we collectively had experience of strong-process projects which weren’t delivering.
FC talked about Agile projects in his organisation. Most of the smaller ones were fine; these were projects with just a handful of developers. But the company’s most significant piece of work was having serious problems. It was now in its third year — nothing necessarily wrong with that, we said, after all some projects are just very large. And what had been delivered so far? Not much. Ah, we see the problem. And yet, he said, to the people on the ground this was a successful Agile project, because they were doing all the right things: daily stand-ups, rigorous unit testing, prioritised iterations… all the paraphernalia of an Agile project. They couldn’t see there was a problem.
AS reported a similar experience. “When I walked into the organisation they proudly walked me through their very rigorous process. ‘But nine out of your ten projects are on Red’, I said, ‘What is that process doing for you?'”
“Is Agile a process?” asked AS, rhetorically, “No. Is there process in it? Yes.” Process helps a project fit into the overall programme, added FC.
BJ went a bit further: successful Agile is about business change, not about what happens in the software team. The Agile approach to a large project is to get just enough clarity to embark on it, which probably isn’t every last detail. The stakeholders need to have the confidence to deal with uncertainty, and that sufficient uncertainty will continuously be resolved as the project progresses.
[And with that I’m going to drop the capital A from Agile unless I’m referring to by-the-book developer-level Agile.]
RN then came in with his take on this. He works in a highly bureaucratic company where technology is driven by heavyweight specs, a focus by high level people on low level details, and delivered entirely by specialist (external) suppliers. Yet his board are continually wanting things delivered faster, to greater quality, and he sees agile as the solution to this. It means persuading his board to think about what they want at the right level — business value, not the contents of drop-down menus — and providing reassurance through clear, regular deliverables. It also means getting his suppliers to deliver in that manner. But all of that is entirely independent of unit tests, story cards, and so on.
I wondered if this meant a top-down mandate of agile was more successful than a bottom up one. AS clarified: it’s successful if it’s business-led, not management-led. LB linked this to Kotter’s 8 steps to successful change, and in particular the first step: there has to be a sense of urgency.
I confessed that our processes in the Guardian team today are substantially less religious (or rigorous) than they were during the time of our major rebuild programme. I used to be able to confidently say we are Agile and point to intensive pair programme, lots of story cards, and so on. Today pairing is inconsistent, automated unit tests aren’t used in non-critical satellite code, and teams don’t always work to iterations. I explained this changed because our business had changed. The emphasis is no longer on transforming our platform but now much more on innovation and on smaller projects with a much wider variety of stakeholders. This makes the control freak within me distinctly uncomfortable (AS had previously commented that in many organisations command-and-control provides confidence) but we generally deliver projects as intended and have happy stakeholders, so I wasn’t going to wade in to change it. BK commented that this seemed perfectly appropriate, and still looked like an agile business.
Successful agile is something that needs to permeate the organisation, from top to bottom. It’s about delivering value, and the vision for what needs to be delivered must be seen and understood at all levels. Many of our experiences of projects going awry were also experiences of where the project vision wasn’t seen by the people doing the day-to-day work.
TC told how his IT department decided it was going to achieve CMMI Level 2. After two years they had accumulated 10 rooms’ worth of documentation, and their productivity had dropped like a stone. And yes, he could measure that drop in productivity, because the team essentially did nothing else for those two years than produce documentation. The board inevitably said: Never do that again. CMMI accreditation may be useful, but it’s got to be for a purpose.
FC talked about his troubled project in its third year. “I revisited the original outcomes document a short while ago, and I couldn’t see how the activities around me are ever going to achieve those outcomes. When I ask one of the team leads what they’ve delivered, and how much there is to go, I’m just waved in the direction of a vast wall of index cards.”
AS talked about a daily scrum he attended: a two-minute meeting in which the team lead got a task update from each team member while he stood with his back to the story card board. “I had to take him to one side and tell him that he needed to be constantly referring people to the project outcomes, not just getting task updates.”
He also told the story of a business manager who wanted a change rolled out midweek — unprecedented for the organisation, as it would involve taking the website down for two hours. This was flatly refused by the Head of Systems. When they discussed the matter the issues were this: the business manager could show that if the change rolled out in the desired slot the company was projected to make an extra £1m, and if the slot was missed there wouldn’t be another opportunity for six months; the Head of Systems, meanwhile, had a target of 99.9% uptime and two hours downtime would mean he’d forfeit his bonus. “Our business must organise itself so that we are all focused on the same goals.”
You may ask what our learnings were. I can confidently say there were no learnings whatsoever because “learnings” isn’t a real word. But there were probably lots of lessons. Here are three key things I came away with.
- Religious Agile, with an optional hothouse atmosphere, is entirely independent of a successful agile project. There are successful projects with and without religious Agile, and there are unsuccessful projects with and without religious Agile. This is underscored by…
- Agile is about business change. It really only works when all the stakeholders focus on the same business-level goals, understand what their part in it is (and isn’t), can deal with uncertainty, and continually work to reduce that uncertainty exactly when necessary.
- The business-level vision must be clear and permeate all levels. This is a corollary to the last point, but to me is significant enough to pull out. The people on the ground have to be able to relate all their activities to the business’s end goal, otherwise they will define success by other criteria — perhaps how well they adhere to a process — which are different from the actual success criteria.
Thanks to all the participants for making it such a stimulating morning.
Lean software development is a fascinating step on from Agile, but its history is in manufacturing cars and to date I’ve only been reading up on lean manufacturing, not lean development. There are two reasons for this. First, I’d feel much more confident about lean if I know the background and reasons for things — in other words, I want to derive lean software development from first principles. Second, I was rather cautious, if not downright sceptical, that something can be translated from manufacturing to software. So if there’s any translating to be done then I want to do it myself — or at least understand someone else’s translation of some of the processes sufficiently to trust the rest of their translations.
This, then, is some of what I’ve learnt about lean working so far, based on manufacturing and with my own projections onto software development. It comes with the caveats that it’s partial and not informed by 99.9% of the work Mary and Tom Poppendieck, the acknowledged leaders in the field of lean software development. But it is informed by my reading of two fabulous books: The Toyota Way by Jeffrey K. Liker (TTW), and Lean Thinking by James P. Womack and Daniel T. Jones (LT). My interest in cars registers on no scale visible to the human eye, but learning about car production from these authors has been quite eye-opening.
In this post:
- Lean challenges common sense;
- Lean is about producing value;
- Lean is about the value stream;
- Lean is all about eliminating waste;
- Lean promises the seemingly-impossible.
The “common sense” way of putting a complicated thing together is to divide the manufacturing process into pieces (the engine, the chassis, the interior, the shell), create a specialist team for each area with their specialist tools, then have them work in a long production line. The result is a very long production line with each product moving from one specialist team to another as it’s built up from beginning to end. If you’re going to be really efficient then you might batch up all the products of one type and send them through in one go, then switch to another product of another type and send those through.
Lean says this is wrong. It’s not trying to be deliberately challenging or quirky or iconclastic just for the sake of making a name for its founders. Rather, it says this is wrong because it generates hidden problems. If we look at the very long production line then every one of the products on that line is in an incomplete state and hiding a multitude of problems. And if you were to implement that single-product-batching it would be even worse.
Lean says the most efficient way of doing something is to create it from beginning to end in a single pass. The single long production line transforms into many, much smaller, teams that are each responsible for the entirety of the production of each product.
This is counter-intuitive. Did we learn nothing from Henry Ford? Well, the evidence — and there is plenty of it — says otherwise.
But I’m getting ahead of myself. Let’s start at the beginning…
This is pretty unstartling and almost uninspiring. But it turns out that what constitutes value is often hard to define. First you have to ask “value to whom?” Often in large organisations people are set targets of something which is not the main goal of the organisation, and this means you end up optimising your processes for the wrong thing. For example, the paint team might be targeted on how quickly they can paint a car body before moving onto the next one. But that would be misleading, and might explain why there’s a retouch team further up the line to cover up the paint problems discovered later. A car company is not about producing painted bodies, it’s about producing cars, and focusing a team on painting ensures they miss the big picture.
The Guardian software team produces software, so it’s easy to see that’s where our value lies and we should optimise our processes around software. But on second thoughts I’m not entirely sure… Perhaps value should be measured from the point of view of the end user. The better we provide news and information the greater the value of what we do, and merely producing software is no good if it doesn’t benefit the end user. And when we do produce software it’s of absolutely no value until it’s released and public.
Once you’ve identified the value you need to map the value stream.
Let’s suppose our value is in producing software that the end user finds useful. The value stream runs from the point of idea inception to the point the user can use it. Mapping the value stream is the painful process of tracking exactly what happens to what and who, and how long it takes. This is painful because you need to be very, very honest with yourself, and map with your eyes wide open. In particular you need to make sure you track all the times something goes back for rework, and all the times the thing just sits there waiting for something to happen to it. It’s also painful because it means managers have to understand first hand what their team have to deal with — anyone who thinks they are above that due to demands of time or priorities is going to find this first step very difficult.
A typical Agile process has a backlog of work cued up before the iteration or sprint, and then hopefully delivered at the end. But there will be large gaps between a story being defined and actual development. If an iteration is three weeks then on average there will be a gap of 1.5 weeks between definition and development starting, and if we insist on any story being up to five days’ effort then a typical optimum scenario is that a single story is at least 60% non-work. This is relevant because…
The primary purpose of lean is to eliminate waste, and I’ve not put it first in the list of things I know because we first need to define value and then map the value stream. Once we’ve done that we can tackle the waste.
Waste comes in many forms — well, seven forms according to lean lore. In manufacturing the most significant of the seven is unused inventory, which means parts that are sitting around in racks and warehouses waiting to be used. Unused inventory not only takes up unnecessary space, it actually masks problems. Here are some examples:
- the acres of spare parts mask the fact that part production is not in sync with the demand;
- the fact that different parts are overstocked by different levels means that it’s impossible to tell which ones are most and least over-produced;
- if a part is only used long after it is produced then quality problems cannot be captured and addressed in time — by the time the quality flaw is discovered there will be many similarly-flawed parts in circulation.
Inventory in manufacturing seems similar to Agile stories that aren’t being worked on: they’ve been specified but are sitting around waiting to be picked off the warehouse rack. And when they’ve been developed they might be sitting around waiting to be released — after all, until it’s released it’s of no value to the customer.
This is where we return to the example above of a story that’s 60% non-work — 60% of its time is just sitting around waiting. The goal is to compress this down to an ideal 0%, but not just because we want to do things quicker. It’s because having sight of something from beginning to end, and not lose sight of it for a second. It’s because being able to focus on something means information isn’t lost, and everyone’s expertise can be brought to bear on it in one pass. If that were to happen then less would be needed to be written down on the story card because the team wouldn’t have to suffer context switching. They could also apply much greater creativity to their work, because they would see exactly why certain things were and were not being specified and contribute alternative or additional ideas without causing confusion or risking repetition.
This idea of taking something from beginning to end in a single pass is called…
In manufacturing this is a big deal. It means rather than having your factory floor as a single production line you create small teams (“cells”) which are responsible for the entire production of each unit (car, lawnmower, etc). If you have huge lathes and paint machines and so on it’s a major change to rearrange the factory floor.
Not so difficult in software development, fortunately — we tend to just have desks and computers, though in any large organisation with centralised functions you always need to win the buy-in of other people.
However, flow comes with a new and serious responsibility for those involved: the cell as a whole is responsible for producing the goods, so they must work together to ensure regular and maximal output. Let me make that concrete…
At no time when I’ve been working with cross-functional teams (software developer, client-side developer, QA, etc) has there been the perfect balance of all roles; we could always have done with one more software developer, or an extra 0.2 of a QA, etc. Much of the time the imbalance is negligible (or quietly welcome), but sometimes it’s very noticeable. And when a cell is working together on a single deliverable (the car, the lawnmower, the software feature) then it’s up to everyone to help each other. It’s no good the client-side developer producing more widgets to test if the QA can’t keep up. They need to ignore the traditional perceived boundaries created by job titles, reach across to others, and work together to regulate the output.
I said lean is about eliminating waste, and flow helps with that in a way hinted at earlier. Flow increases quality by allowing all participants to see the thing put together from being to end without interruption. This reduces hand-off time, reduces information loss, reduces relearning, and increases knowledge and ownership. The number of times an item needs to go back for rework is reduced, and if it does need to go back then the rework that’s needed is clear and therefore quicker.
Meanwhile, all this work needs to come from somewhere and that’s what “pull” is all about. The principle of lean is to only do what’s needed, and that means only produce something that is a direct response of a specific request, and only when needed.
The distinction is clear in car manufacturing. A car company’s marketing department will devise a special offer on a particular configuration (these seats, those mirrors, any one of the following colour combinations) and the plant will have to manufacture a whole lot of those particular cars ahead of time, but only with a guess (rather than a certainty) about what the demand will be. In the lean world a car is only produced in response to a specific customer’s specific purchase: customer goes into showroom, customer orders car, order triggers build.
That’s how pull works in relation to delivering the product to the customer. But pull also works in relation to building the product inside the factory. The old method is to keep hundreds of each part in store; the lean method is that a part is only provided to the worker when they need it: when they’re running low they signal the need for more, and it’s provided for them. This triggers a chain right back potentially to the supplier of the part, ensuring they are always delivering just enough, no more and no less.
The parallel flawed system in the software world is the product backlog. (We’ll ignore the even worse scenario of waterfall’s detailed planning up front.) Work is prioritised ahead of the sprint and waits to be developed. Requirements can change, even in that gap between prioritisation and development. The consequences aren’t as terrible in the Agile world as in the waterfall world, but it still causes problems: it disrupts the team’s schedule and of course all the effort that went into the planning of the now-deprioritised story is wasted. Even if the requirements remain constant the gap between planning and developing mean knowledge is lost, or conversations need to be repeated, or the requirements turn into mini-waterfall-style requirements specifications.
The lean software alternative is to prioritise the next story only when the team is ready to work on the next story. That means while the team is developing the current feature they don’t have much certainty about what’s coming next. Like the worker in the car plant they have to signal slightly ahead of time that a new story needs to be worked out. Then the Scrum Master/business owner/internal customer needs to get something ready so that when they become available the team can all get together, thrash out the details, and set to work again.
As a manager I’m uncomfortable with this: I can no longer know what’s going into an iteration when the iteration starts. But the definition of value is not “what a manager’s comfortable with”. I will need to find other ways to ensure we can be accountable to the rest of the business, and they will need to be ways which are closely aligned with end user value — and that can only be a good thing.
The combination of flow and pull doesn’t mean the team is only working on one thing at a time. But it does mean that everyone has an equal balance of work at all times. So if Alf’s development work gets passed over to Betty’s testing then Alf and Betty need to make sure that she is expecting to finish her current piece of testing at pretty much the same time as Alf finishes his current piece of development and passes it over to her. Keeping that flow even is really important.
Lean holds out the seemingly-impossible promise of increased productivity and increased quality. But here are some numbers from the literature:
- The Puget Sound Naval Shipyard, time to prepare a repair document. Originally: 97 days. After a lean workshop and taking action: 26 days. (TTW, p.103)
- Lantech wrapping machinery, before and after lean. Production throughput time was 16 weeks, became 14 hours to 5 days. Delivered defects per machine was 8, became 0.8. Employee time per machine was 160 hours, became 80 hours. (LT, p.121)
- Porsche, production of the 911, before and after lean. Initial stamping to final shipping was 6 weeks, became 5 days. Inventory held was 17 days’ worth, became 4.2 days’ worth. Errors from work on the assembly line dropped 55%. (LT p.213)
I do have some concerns about lean software development, but they’re less about lean itself and more about bandwagon-jumping and doing things without really understanding the reasons. Regardless of that, it’s refreshing to find a new way of looking at what seem to be known problems, and making insights which you might not otherwise have found. It’s certainly something I’ll be spending quite a bit more time on.
…which was a semi-regular event of Wii tennis in the office, but a very useful part of our R2 work, too.
Each launch required a small army of technologists to be on-hand: to run the various scripts, check the results, and deal with any problems that might arise. We needed to arrange these teams carefully because launches happened overnight, so we’d need an overnight team and another team in early the next day to pick up any remaining issues.
All this was fine, but after a while someone realised we’d missed a trick. I think he actually wanted to be part of the overnight team (it’s always exciting to see these thngs go live) but wasn’t actually on it. So he arranged with others to bring in some games consoles, and wired one up to a big screen, the others to projectors. It was a big draw, and a great way to have an extra group of techies staying late in case the need arose.
Still, we couldn’t be cavalier with this. For example, we made sure the gamers were located far from the launch team — the launch team had a serious job to do and didn’t need distraction. But that didn’t prevent the benefits: every so often a gamer would slip out of Guitar Hero and wander over to the launch team to check up on progress, offering some advice and support if necessary. One time a critical SQL script was running worryingly slowly and a call came through to see if someone could contribute to the investigation; a couple sat down at a machine a few feet from the Mario Kart players, traced the problem, and suggested a change to the script which was agreed and did the job. That night it made the difference between “go” and “abort”.
If there’s a general lesson to be drawn from this I don’t think it’s to keep a Wii console in the stationery cupboard along with the paperclips and envelopes. But exploiting opportunities that are specific your particular situation is probably a good thing to do, even if they aren’t enshrined in official company policy.
…which was the penultimate step before a launch, after the software had been built and released, and before the technical work to finally lift the curtain.
One of the big changes that was part of R2 was how we structured our content — our information architecture. Previously each piece of content lived in a section, up to two levels deep, and a lot of content was duplicated so that it could appear in more than one section at a time. An extreme example we often used was the affair around David Kelly and the consequential Hutton inquiry. Almost every story there crossed the boundaries of politics, media and daily news.
With R2 we were introducing much more nuanced keywording and more options around navigation. So the content in the old system didn’t map directly into the new system: it all had to be examined and reclassified by hand. Additionally, production staff needed to build subject pages in ways they couldn’t before — for example, the pages on Afghanistan, the British monarchy and the BAE corruption investigations. All this was called sitebuilding.
Of course, the tech team built tools and wrote scripts to make the production staff’s job easier, but some things just need human expertise and take a very long time. Typically we allowed six weeks between the time the software was released and the relevant site was launched and that was the period in which sitebuilding took place.
…which is a much misunderstood subject.
The R2 team consisted of a number of QAs, and the most obvious artifact that the QAs produced and worked with was the test script: a series of detailed instruction that explained what to test and how to test it. For this reason it’s too easy to dismiss QAs as testers, and that would be a mistake.
Our QA Manager, Amy, says “I don’t want us to be finding problems, I want there to be no problems to find”. That hints at how QAs should be used: they should be involved from the very start of a piece of work to identify an appropriate structure and approach that ensures greater reliability and more opportunities for testing. In the best cases a QA has guided a task in a direction that BAs, developers and architects might not have previously considered, so avoiding problems they hadn’t thought of. In the worst cases a QA has been omitted from planning a piece of work and something considered straightforward by the QA-less group has turned out expensive to test and a constant source of problems.
Testing comes at the end of the development process, and considering a QA as a tester therefore allows one to fall into the trap of including them only at the end. The quality assurance process should add value, and that can only happen if the QAs are involved from the start.