This is third in an accidental series on testing, and today I’m going to walk through a thought exercise in improving test times. This follows directly from last week’s post about the 10 second build, and challenges the assumption that end-to-end functional tests are essential.
You’ll recall that Daniel Worthington-Bodart asserted that software build times could be dramatically reduced through better software design and more thoughtful tests. And “more thoughtful tests” means unit tests replacing extensive integration and functional tests. But I think most of us who have been involved in software testing have seen plenty of times when end-to-end functional tests seem essential and unavoidable. Taking on Daniel’s challenge I thought back to one such situation and tried to ask myself if perhaps the end-to-end test could have been avoided…
We had a system in which an ID was stored in a database as a string. It was pulled out of the database by a back-end service, was passed through transparently, and then consumed via an API call by a front-end application. Then one day the front-end application broke. The bug was traced back to the ID string format having changed: the database was agnostic about the format, the back-end service passed the string through transparently, but it turned out the front-end application was depending on it being in the original format.
The fix was relatively easy: update the front-end application. But then the conversation turned to testing. We should have caught that before it happened, we said. We should have an automated test which caught it, we said. The only automated test which could genuinely have caught that is an end-to-end functional test, we said. After all, we said, you can’t fault the database, you can’t fault the back-end service, the front-end application was just responding to what it had always expected, and you can’t expect every change in one part of the system to be communicated and its impact understood by teams working on other parts of the system. Only an automated functional test would have caught this. And we all nodded our heads, and went off to put more effort into functional testing.
But having heard Daniel’s case for the 10 second build I wondered if we were giving up too easily. What would he have said? I think he would have said a couple things: (i) The contracts between the various layers are not clearly stated, and (ii) there is no consideration of what happens when a contract is violated. Let’s take those points in turn…
First, a system’s outputs should have clear constraints. Is the outgoing ID supposed to be an opaque string, or will it have a particular format? This should be stated, or documented, in some way at each level: the database level and the back-end service level.
Second, those contracts should be tested. What happens if someone puts an ID into the database of the wrong format? What if the back-end service receives an ID in an unexpected format? What if the front-end application gets an ID in an unexpected format? Can a component or function deal with an empty string, a string with spaces or international characters, an over-long string, etc? Any of these things can have an automated unit test or a limited integration test which is not a full end-to-end functional test.
One of the reasons software goes wrong is untested assumptions which change: the assumption that a file will always be there, the assumption that the app will always run on version 10.4 of the operating system, or the assumption that a string should or should not be in a particular format. Arguably the problem we faced was not “something unexpected changed” (which might have necessitated a functional test) but rather “we did not clarify what data our systems were supposed to output” and “we did not clarify what our systems were able to consume”. One system made an assumption it didn’t test; another system allowed that assumption to be made.
If a test was being written for the front-end app for a specific ID data format then it should have prompted the question “what is the right data format?” which would in turn have forced the developers of the other layers to clarify their own systems.
This has been a rather lengthy discussion of a relatively small problem. But I think it’s a good example of exposing unclear thinking through a rigorous approach to testing.
niksilver.com 
Very good post, Nik. It’s obvious, but your example also highlights either a key missing acceptance criteria or missing tests to confirm that the AC had been successfully met.
I’m sure with hindsight it’s possible to construct some low level tests that would have found this problem, but I’m sceptical that anyone can think of everything that might go wrong in advance and design tests for it all. In practice, some simple, happy path functional tests are the only thing that gives me real confidence.
@Simon – If happy path functional tests are the only thing for you, then I admire you for being a wholly happy person ;-)
Regards being able to think of everything that might go wrong in advance, yes, that’s certainly as-good-as impossible. But I don’t think I was claiming one can foresee every possible problem. Rather, I was attempting to show (i) more rigorous thinking could have revealed a lack of contract agreements between our systems; (ii) rigorous unit testing could well have revealed this lack of contract agreement; and (iii) although we thought the only possible reliable regression test was an end-to-end functional test, when forced into it we could have come up with a faster, less elaborate alternative.
I would reluctantly concede that a real 10 second build, with no end-to-end functional tests might be an impossibility for a real-life project. But it’s such an attractive idea, and the possibility forces us into new insights that we would otherwise have given up on (as it did for me above), that I think it’s definitely worth striving for. A bit like .
I completely agree with you, Nik, that developers need to get into the habit of rigorous thinking about their tests. It is very easy to build in tests that cover the scenarios about which you know or have had specified, but it takes real thought to build tests that trap the unforeseen or that catch changes in base assumptions. And as you rightly observe, where you have different teams working on different parts of a system, it becomes even more important that people design from the outset to cope with the hand-off points between application layers.
But what I find challenging is legacy systems built with little or no embedded tests. Retro-fitting is a nightmare to do and difficult to make rigorous – it is far too easy to write in tests that prove that the existing code works rather than test that all the conditions in the original spec (and any post-release changes) are being met. Also in many legacy systems you don’t get the well-structured touch points your case study had – you often have to deal with multiple entry and exit points, each of which could have been developed by different people at different times using different methods (and possibly wearing different cowboy hats).
Is that not an instance where broad, if not end-to-end functional testing has an inevitable place?
@Tom — Undoubtedly legacy systems are challenging, and they rarely allow smaller testing without extensive rework. Indeed, one of the points about the 10 Second Build is that it fosters good design, with the corollary being that good design is less likely to be there if build time (including testing) was never considered.
The only place I’d differ with your view — and it’s only slight — is the word “inevitable”. I can’t help feeling that has a hint of “giving up” about it. I’d much rather the approach was “we’ll balance the options, we’ll choose what’s right for the present, and we’ll never lose sight of how to make things better”. But in reality, yes, end-to-end functional tests might be the most appropriate thing for the circumstances.
No need to differ – my use of “inevitable” was only meant to indicate a response to what you might find on your plate at a given moment. Without doubt the realisation that the only viable choice is to fully test the entire system for pretty much any change should trigger a knee-jerk reaction that change is essential. Whether it is refactoring or replacing, something remedial is almost certainly warranted – assuming the legacy system is of material importance.
Incidentally, my reaction to Daniel’s video talk was moderately interesting right up until the last couple of minutes. That was when he made the link between the tests (and their value) and the quality of the application design. To me, that should maybe be the starting point when you realise your tests and builds are taking a long time – is it indicating that your application design might actually be the underlying cause of the slowness? Are you doing a lot of unnecessary repetitive work? Can you cache data rather than retrieve it fresh every time? Are you recalculating values despite knowing they won’t have changed? If you have built tests into the code that does that redundant processing, running the same tests repeatedly (and unnecessarily) will certainly hit your build times.
It just occurred to me that NOT revisiting the underlying code and just concentrating on speeding up the tests could actually mean you mask a fundamental problem. You might end up making it quicker to build and release code that is less permformant or scalable than it might be if you took the hint from slow tests or builds and looked at the whole rather than just the parts. Now I get his comment, “Treat your tests as production code”.