This is third in an accidental series on testing, and today I’m going to walk through a thought exercise in improving test times. This follows directly from last week’s post about the 10 second build, and challenges the assumption that end-to-end functional tests are essential.
You’ll recall that Daniel Worthington-Bodart asserted that software build times could be dramatically reduced through better software design and more thoughtful tests. And “more thoughtful tests” means unit tests replacing extensive integration and functional tests. But I think most of us who have been involved in software testing have seen plenty of times when end-to-end functional tests seem essential and unavoidable. Taking on Daniel’s challenge I thought back to one such situation and tried to ask myself if perhaps the end-to-end test could have been avoided…
We had a system in which an ID was stored in a database as a string. It was pulled out of the database by a back-end service, was passed through transparently, and then consumed via an API call by a front-end application. Then one day the front-end application broke. The bug was traced back to the ID string format having changed: the database was agnostic about the format, the back-end service passed the string through transparently, but it turned out the front-end application was depending on it being in the original format.
The fix was relatively easy: update the front-end application. But then the conversation turned to testing. We should have caught that before it happened, we said. We should have an automated test which caught it, we said. The only automated test which could genuinely have caught that is an end-to-end functional test, we said. After all, we said, you can’t fault the database, you can’t fault the back-end service, the front-end application was just responding to what it had always expected, and you can’t expect every change in one part of the system to be communicated and its impact understood by teams working on other parts of the system. Only an automated functional test would have caught this. And we all nodded our heads, and went off to put more effort into functional testing.
But having heard Daniel’s case for the 10 second build I wondered if we were giving up too easily. What would he have said? I think he would have said a couple things: (i) The contracts between the various layers are not clearly stated, and (ii) there is no consideration of what happens when a contract is violated. Let’s take those points in turn…
First, a system’s outputs should have clear constraints. Is the outgoing ID supposed to be an opaque string, or will it have a particular format? This should be stated, or documented, in some way at each level: the database level and the back-end service level.
Second, those contracts should be tested. What happens if someone puts an ID into the database of the wrong format? What if the back-end service receives an ID in an unexpected format? What if the front-end application gets an ID in an unexpected format? Can a component or function deal with an empty string, a string with spaces or international characters, an over-long string, etc? Any of these things can have an automated unit test or a limited integration test which is not a full end-to-end functional test.
One of the reasons software goes wrong is untested assumptions which change: the assumption that a file will always be there, the assumption that the app will always run on version 10.4 of the operating system, or the assumption that a string should or should not be in a particular format. Arguably the problem we faced was not “something unexpected changed” (which might have necessitated a functional test) but rather “we did not clarify what data our systems were supposed to output” and “we did not clarify what our systems were able to consume”. One system made an assumption it didn’t test; another system allowed that assumption to be made.
If a test was being written for the front-end app for a specific ID data format then it should have prompted the question “what is the right data format?” which would in turn have forced the developers of the other layers to clarify their own systems.
This has been a rather lengthy discussion of a relatively small problem. But I think it’s a good example of exposing unclear thinking through a rigorous approach to testing.