I don’t normally find myself in conversations about over-engineering, but the subject came up twice last week, and that gave me a chance to think about the issue. Senior execs and project managers can be worried that developers over-engineer software, thus costing them unnecessary time and money. Undoubtedly it does happen. So, why?

The first thing to clarify is that developers almost never set out to over-engineer something. Over-engineering is, by definition, excessive engineering, and only the most peculiar developer will deliberately create excessive complexity.

This means over-engineering is really a difference of perception. The developer and the observer (who may be a project manager, another developer, or the same developer some time in the future) have different views of what is appropriate. With this understanding the question of over-engineering stops being “Why did they engineering excessively?” but rather “Why are our views different?” I think there are three answers.

First it’s a problem of communication. A requirement should not be “I want this…” but rather “I want this so that…”. The “so that” needs to capture the intention of what the thing will and will not be used for. If that information is not communicated or not understood then there will be a mismatch in what is believed to be an acceptable solution.

For example, suppose a project manager wants a box for text entry and the developer creates a box which allows visual formatting like bold and italic, which the project manager deems excessive. This may well be due to a failure of communication. Perhaps the project manager believes the data in the box will only ever be used for simple, raw text, but the developer either has not heard that limitation or has failed to understand it. Perhaps the developer knows of a further requirement down the line where visual formatting will be introduced. If the two have not discussed this — will it be needed with certainty? Do we expect to have time to add it later? — then again there will be a mismatch in perception about what is acceptable now.

Second, it can be due to a problem of trust. The project manager may say a simple solution will suffice, and they may say there will be time to revisit it later, but the developer may not believe this, and therefore choose a more complex solution. Trust can be won over time, but it’s difficult, and it requires both parties to be honest, to keep promises, and to allow the other some freedom.

Third, it can be due to an honest error of judgement. This is particularly the case when a developer looks at their own work several months down the line and sees over-engineering. That kind of problem is more forgiveable. But even that situation can be reduced with appropriate tools, architecture and processes that allow rapid extensions later if needed, on top of a simpler initial solution.

So over-engineering is a genuine phenomenon, but it’s not just a simple case of the developer doing a bad job. It’s something that can be improved by work from both parties, primarily through improved communication.