Regards (b), be aware that my low-tech survey followed on directly from a director asking me to demonstrate that the team was improving. If I’d have gone in with that survey technique but the director had really wanted to improve revenue then it would have been the wrong tool to apply.

So that takes us back to (a), making sure the questions are meaningful. It’s important that when we’re asked to measure a change that there is a purpose to the measurement, and that we can perceive a difference in the world with and without that change successfully applied. As long as we can perceive a difference then we can describe it and measure it.

As a rule of thumb I’d say software metrics are less important than real-world metrics (if that’s the right thing to call them). Software has to be about something. Brilliantly designed software that doesn’t help the business is less useful than poorly designed software that does.

Someone like an Engineering Director will probably be concerned with software metrics. In that case code coverage and method sizes are perhaps meaningful. But there’s no meaning beyond the engineering department. Those metrics are pointless to the Commercial Director.

As you said, there are many metrics out there that we could use in order to “measure” the benefits.
But it seems to me that really none of them are able to analyze the whole system, but tend to look rather at its sub-parts, and that’s why I see them as being more or less misleading.
One does not have to dig very deep in order to find the “right” metric that proves his point of view, whichever that one might be, good or bad.

The low-tech technique you’ve described above is among my favorites methods as well.
I like it mainly because it is simple and it looks at this complex system in a holistic way.
And really, who else beside the team members and the stakeholders know better “how are we doing”?

Still, I find it very difficult to make others (read: upper management) understand and really buy into the outcomes of such a survey. Some regard it as being only anecdotal and think of it being very soft, because it relies on subjective opinions, rather than relying on hard data like number of defects or test code coverage.

Have you ever had such issues?
If yes, how did you dealt with them?

The software was being developed with a visual tool, in which the data flowed through various steps, laid out on a canvas, and joined by connecting arrows. Thus the tool presented a visual representation of the logical flow. My drawings above give an example of what the tool showed — and thus what the logic was — in each scenario.

However, messing thinking leads to messy software design. And the messy software design in the first version was quite apparent just from looking at the logic laid out on the canvas. That’s what point #4 is saying.

I’m not making any kind of comparison with non-visual tools, BTW.

And thank you very much for your generous comments

Good job

On your more serious points, any change can cause damage if it’s not properly understood.