Over on the emergn Value, Flow, Quality blog Chris Berridge says

If the estimate information you are producing isn’t vital for making a decision, then take a good hard look as to whether you need it. Some of the teams I have worked with have stopped some kinds of estimates as these estimates were not changing the decision.

But often it’s not a binary choice of do we/don’t we need the estimate. Instead it can be a case of “we need to know if it’s likely to be more than…”. In this case producing the estimate is about determining where the project sits in relation to that threshold. That means estimation is still valuable, but accuracy may be more relaxed.

Recently I was involved in what I consider to be a good estimation exercise. The team was asked to estimate a project, we came up with figure, and the stakeholders consequently decided that the project was too big and should drop down the priority list. I regard that as a success because the number was not questioned and it did influence a business decision. But some of the developers involved were unhappy with the exercise because they didn’t know how literally their number would be taken, and felt that what they were producing was going to be turned into a deadline. I’m sympathetic with that perspective: it was a stressful experience despite a positive outcome.

This is a difficult situation to fix, though. We would have had a much less troubling time if the stakeholders had said at the outset what “too big” would have looked like; then we could have estimated only if the project was likely to be bigger or smaller than that figure. But in reality the stakeholders didn’t know what “too big” was, so couldn’t have communicated it. It was more a case of “we’ll know it when we see it”.

In the end someone has to put their cards on the table first, and usually that’s the tech team. The stress can be removed from the situation if, over time, it is demonstrated that those tech estimates are treated as intended — not as promises, but as contributions to informed business decision-making. The stakeholders demonstrate their respect and understanding of the development process, and the development team sees that they are regarded as genuine partners in the business.

An API is valuable in its own right because it streamlines development significantly. The communication with the core development team and amount of integration code that needs to be written both drop dramatically. I’ve been in the unusual scenario where an application had to be written twice from scratch, once when there was no API and once when there was one, which makes for a good A/B test. First time around the application effort matched the integration effort. Second time around the integration effort dropped dramatically — from months to days.

But an API really comes into its own when you use it yourself, internally. For one thing it forces you to make sure it’s really usable; you will flush out a lot of issues and make it more relevant to external developers before they encounter issues themselves. For another, you can expect those integration cost savings to be reaped internally: your own developers will find it much, much easier to build on your platform, and therefore you can cut through that impossibly-long to-do list at a much slicker pace.

None of this is free of course. Like so much, it requires some kind of investment. The API won’t be right first time, so it involves learning and feedback. If you have a version 2 and are considering moving people from the old version then you need to choose between migration costs, maintenance costs, or dropping those older services.

But once you have made those investments a new world awaits. And all your problems will be… of a better class.

The article was interesting in other ways, too. Its headline might have suggested that Facebook’s designers might be playing the role of lead band member and Smash Hits pin-up, the content was more measured — it was really about how they are essential to the team. That difference of headline/content is a good example of how we think. We all want easy answers — “how does Facebook do it?” — and the rockstar designer is a good example of the easy answer — “Of course! They make design central; we should hire some rockstar designers of our own, and then we’ll win like Facebook”. Reality, though, tends to be more nuanced and complicated, and creating cults around individuals or individual functions is dangerous.

There are frequently stories in tech circles about how staggeringly valuable it is to find the right developers. and the Reuters article says “Entry-level interactive designers at startups are commanding salaries easily topping $80,000, almost twice the median pay for primarily print designers of about $45,000″. I know one entrepreneur whose philosophy was to find a brilliant, if anti-social, developer and let them get on with it while his managers slid pizza under the door. I know a business team who let their product development process be driven by a brilliant designer, largely alone.

But it’s important to avoid cults — the cult of the designer, the cult of the developer rock star, the cult of the brilliant city trader… It’s much more valuable to build a fantastic team.

People in a company come and go. Putting one of those people on a pedestal might be great for the short term but it creates a single point of failure and it’s dangerous for the long term. If that person has a mission then if they leave before it’s done the people who remain are rudderless. If that person stays beyond their mission completion then the value they are adding from then on is questionable (and aimlessness is damaging to their personal development). Either way, letting an individual or a group of individuals operate unchecked is very likely to lead to problems.

The pizza-sliding entrepreneur went onto greater things, but the technology he left behind was a mess. The business team found they couldn’t align the rockstar designer’s work with everything else they needed to do and had to let him go. And we know what happens to investment banks who let their traders operate unchecked.

Cults are dangerous in business. It’s the team that counts.

The background

Victoria Song, former VC, wrote about her experience working in a startup — actually, as the first person in its NY office. She expressed her perspective very effectively in the final sentences of her blog post, and I think it captures the feelings of many I know in similar situations:

…I’ve never felt more energized. I’ve also never felt more connected to my work. There was a moment I had doubts that I’d be able to make the impact I want in only 6 months, but boy was I wrong. 1 day in startup life is equivalent to 200 days in a big company. Shit just gets done.

That’s great! And I work with a couple of startups, so I know it’s true. Working in startups is great. And…

Bigger is beautiful, too

One of the reasons I love working inside bigger companies is that they’re just more complex. There are more people, real revenue streams, long-standing relationships, talented specialists, and more. If you want to make change (and from where I tend to stand, change is the norm) then you can’t do it by yourself, you have to do it with other people. And you don’t have any choice about which other people they are — they’re the people who already work there.

On the one hand that makes the job harder. On the other hand, it makes the achievements greater.

I can code, but I can achieve more with a professional software developer or twenty. I can think through actions, but I can achieve more with half a dozen professional QAs. I can co-ordinate things, but I can do it much more effectively with a project manager or two. It’s no longer down to me — it’s down to us. I need to justify things not just to myself, but to other real people with their own perspectives and their own commitment to the organisation. And often their perspectives are more valid than mine, and when the stronger idea is recognised and wins out we become more than the sum of our parts.

All this takes more time, it’s true. Working with colleagues in a real business requires empathy, a bit of psychology, open-mindedness, a commitment to long-term relationship building and respect. I like to think I’m pretty effective at my job; but I know I’m more effective with help.

I couldn’t help marvelling at how things have changed. Three years ago, when we launched the Guardian’s content API, it seemed a pretty radical idea. We subsequently architected several other applications with internal RESTful APIs, even though we didn’t always have concrete plans for any other clients; it just seemed sensible. Last November in Bucharest I saw my friend Alan Bradburne talk about architecting with RESTful APIs. What struck me there was that he made it seem the most natural thing in the world — “Why would you do it any other way?” he seemed to be suggesting. And now here was someone with a no technical background using it in the vision for his company.

It also took me back to the early days of the web, when web-browsers-as-clients was still a new idea. I would visit blue chip companies and talk to them about the wisdom of three tier architectures. My audience would be made up of people much more senior than me, but they would still listen closely, nod at my words, and write things down.

Today if you spoke to people about three tier architectures they wouldn’t be remotely interested; there’s no insight there today. Platforms and platform APIs are not quite at that stage yet, but they will be very soon. Even when designing a system for internal use they are valuable, and you don’t have to be an organisation as large as Amazon to appreciate that, although they set a pretty good example.

Meanwhile I think there’s a progression here. There’s another architectural pattern just around the corner that’s next in the sequence. It may have something to do with the unstoppable force of BYOD — bring your own device. Or it may be related to a trend that we can’t yet see. Answers on a postcard — or a whiteboard — please…

Like many technologists, I often find myself in difficult organisational situations that I think I know a way out of. That’s because lots of difficult organisational situations involve technology, and because we technologists are good at sharing ideas, so it’s understandable that we think we have a solution to situations we understand.

But having a solution and implementing that solution are two entirely different things. It’s too easy to attempt to implement a change only to get push-back for reasons that may or may not be clear.

Thus it’s particularly important to understand people’s point of view when they perceive a problem. That involves extra time in speaking to them, before rushing to action, which can be annoying — after all, planning is the boring bit; doing is seen as the fun bit. So how can we get the most out of those initial exploratory conversations as quickly as possible?

One technique I’ve found valuable is to ask about behaviours. Some examples:

• “What do you think about X?” followed by “What have you seen that makes you say that?”
• When someone attributes a motivation to someone, asking what they’ve seen and heard that leads them to think that.
• “What would you like to see changed?” followed by “What would you expect to see people doing differently if that happened?”

I find the answers to these questions tends to transform my perception of a situation and any personal interpretation I might have added without justification. It can immediately reveal that the perceived problem is either not there or is actually a different problem — the interviewee having jumped to a conclusion. You can use the opportunity to explore that with them. It can simplify the problem, bringing it down to something concrete and tangible. It can also shape a solution, clarifying what success looks like.

There are lots of tools for enabling constructive conversations in difficult situations, but I find most of them are too complicated to remember in the heat of the moment. A very simple one like this, asking about behaviours, is one I can remember and it yields good results.

I find Scrum and kanban have a lot of overlap: small pieces of work, described as cards on a board, moving between swimlanes. A group stand-up once a day, someone to walk the board, an aim to deliver consistently.

The core difference is that Scrum has a batch approach: a commitment to deliver a predefined set of items in a timeboxed period… and kanban has a flow approach: limited work in progress at every stage with the aim of delivering continually.

Kanban also mandates that individual user stories are timed from beginning to end. Scrum does not have this mandate — instead we are more likely to measure how many user stories are delivered in each sprint. Ideally it will be at least the number we committed to.

Because kanban has a focus on individual stories, not batches, I find problems are surfaced much more readily — you have far fewer options for moving something forward, so if there is a problem then it’s harder to avoid. Indeed, sometimes it’s the only thing you can possibly work on. By contrast I’ve seen plenty of times in Scrum where story cards bunch up in one swimlane, which is a clear sign of a problem, and yet the team avoids seeing it because they can get on with other work. And I’ve seen times where work doesn’t get finished in an iteration, consistently. Again it’s a clear sign of a problem, but too often the team just says c’est la vie, and continues as before.

You may say that such bunching and overhang and general ostrich-like activity is due to a lack of team leadership, and I’d be inclined to agree with you. But I think the nature of Scrum makes it easier to take that approach.

When I think of kanban I think of slowly lowering the water level to reveal the rocks — the problems. This works because of its “flow” nature, and its continuous push for delivery. Scrum does not have flow, but rather canal locks or sluice gates. It does reveal problems, but in a different way and (I think) less readily.

A couple of footnotes. First, you should read the original post which spawned my conversation with Dan — he was trying to capture the difference between Scrum and kanban in one word.

Second, in all of this I’m reminded of Rachel Davies’s comment on the recent IT Kanban podcast. She commented that each seminal book on a development practice is written only on the basis a few cases studies that were the experience of the authors, because they were — by definition — the earliest adoptors of that practice. Since then the world has changed and the practices have evolved in myriad ways to address the individual experiences of the others who have adopted them. Similarly my views above are inevitably based on my own experiences and uses of the practices. I’m sure others have different views based on different experiences. I’d love to hear them — particularly if you see Scrum and kanban differently.

The first thing to clarify is that developers almost never set out to over-engineer something. Over-engineering is, by definition, excessive engineering, and only the most peculiar developer will deliberately create excessive complexity.

This means over-engineering is really a difference of perception. The developer and the observer (who may be a project manager, another developer, or the same developer some time in the future) have different views of what is appropriate. With this understanding the question of over-engineering stops being “Why did they engineering excessively?” but rather “Why are our views different?” I think there are three answers.

First it’s a problem of communication. A requirement should not be “I want this…” but rather “I want this so that…”. The “so that” needs to capture the intention of what the thing will and will not be used for. If that information is not communicated or not understood then there will be a mismatch in what is believed to be an acceptable solution.

For example, suppose a project manager wants a box for text entry and the developer creates a box which allows visual formatting like bold and italic, which the project manager deems excessive. This may well be due to a failure of communication. Perhaps the project manager believes the data in the box will only ever be used for simple, raw text, but the developer either has not heard that limitation or has failed to understand it. Perhaps the developer knows of a further requirement down the line where visual formatting will be introduced. If the two have not discussed this — will it be needed with certainty? Do we expect to have time to add it later? — then again there will be a mismatch in perception about what is acceptable now.

Second, it can be due to a problem of trust. The project manager may say a simple solution will suffice, and they may say there will be time to revisit it later, but the developer may not believe this, and therefore choose a more complex solution. Trust can be won over time, but it’s difficult, and it requires both parties to be honest, to keep promises, and to allow the other some freedom.

Third, it can be due to an honest error of judgement. This is particularly the case when a developer looks at their own work several months down the line and sees over-engineering. That kind of problem is more forgiveable. But even that situation can be reduced with appropriate tools, architecture and processes that allow rapid extensions later if needed, on top of a simpler initial solution.

So over-engineering is a genuine phenomenon, but it’s not just a simple case of the developer doing a bad job. It’s something that can be improved by work from both parties, primarily through improved communication.

And I know plenty of good recruitment agents, all of whom dislike them. That’s because a good recruitment agent can identify good candidates and sell your company if they understand the subtleties of your operation, and they can only do that if they get to speak to the human beings who are doing the hiring. If they can’t speak to human beings then your company and that role is hard to distinguish from any others on their books, and so you won’t get many good candidates.

I know HR departments defend recruitment portals, but HR departments rarely have a direct interest in the process. Despite their difficult position (responsible for the integrity of the hiring process, but without the authority of actually doing the hiring) they only have a periphery interest in either the speed of the appointment, the happiness of the candidate, or the consequent rightness (or wrongness) of any final decision.

A game of three thirds

At this point I should admit to some acceptance of recruitment portals and make the argument a bit more nuanced. There are three things mixed up here: the fact of a digital recruitment system; the usability of that tool and its place in the process; and the requirement for all activity to go through the tool, preventing people from direct communication. To take them in order…

It’s fine to have some kind of database or digital system for collecting and storing applications — that’s just good organisation. If CVs or applications come in electronically then you’ll want some way of tracking them. That might be an e-mail inbox (possibly with folders), a spreadsheet, or something grander. Storing things electronically is fine and usually appropriate. And if you’re operating in a very large organisation with hundreds or thousands of applications every day for essentially the same kind of jobs (software engineer at Google, entry-level accountant at PwC…) then you’ll certainly need something a bit more than a spreadsheet to manage them.

The usability of the tool, and its place in the process, is slightly different. If your recruitment process is finely honed, and all your hiring managers are known to operate in exactly the same way (have exactly the same requirements, sort candidates in exactly the same way, look for exactly the same things, assess people in exactly the same way) then you can design and configure your tool to be just right for all those managers. But that’s very rarely the case. Most people aren’t hiring managers in Google or PwC looking for hundreds of the same person in the same manner. Most hiring managers at the bottom of the chain (i.e. the ones who are a primary filter before finding a candidate good enough to put in front of their boss) have only a couple of dozen applications to manage at any one time, for one or two positions. Those of us in that position want to focus on the people, and a tool that’s not exactly in line with the way we work is just going to get in the way. That’s particularly frustrating when we know we could do a lot better with a spreadsheet or an inbox. To HR people reading this asking “Is it really that bad?” the answer is Yes, bad tools really get in the way.

Finally to the issue of preventing hiring managers from speaking directly to recruiters, the corporate equivalent of “talk to the hand, ‘cos the face ain’t interested”. This should have nothing to do with recruitment portals — after all, using technology and speaking to people are entirely compatible — but that’s rarely the case. The recruitment portal is too often elevated above its station, and becomes the entire process rather than an aid to the process. Eliminating human contact from an essentially human endeavour is a major failure if not a tragedy. It’s forgetting the human and thinking only resources. Job hunting and hiring is corporate dating, and personal contact is key. As I said above, if you can’t distinguish yourself through personal contact then don’t be surprised when your candidates are disappointing — the good candidates will be talking to your nimbler, less bureaucratic competitors.

Premature technologification

Looking at the sales blurb for recruitment portals I see they tout cost savings and administration efficiency. But other than the mass-intake examples I cited above, most hiring is not a standard process that can be mechanised. Thus prioritising cost- and efficiency savings in a variable process damages the core purpose: to hire the right people for the job. This is an example of premature technologification.

Different jobs have different approaches to hiring (for some an application form is standard; for some it’s an anathema); different managers prioritise in different ways; HR’s needs are different from hiring managers’ needs; hiring context changes. You can’t keep configuring and reconfiguring a portal every time, so you end up with a technology that doesn’t fit the situation.

The humanity needs to be central to hiring, otherwise any cost savings that might come from the recruitment portal will be dwarfed by a new and more significant cost — the cost of not getting the right people into your company.