aelena.com

Software testing is routinely underplayed, ignored and underestimated when planning a software project. Testing is not a single thing, rather it consists of different tasks, most of which take more time than many seem to think. Sadly, it’s quite common to give little thought to these tasks, as well as the risks associated with each. The temptation to cut corners on testing seems to be irresistible, even when paying that tax later on hurts like hell. So what is testing made up of?

First, somebody needs to come with an adequate selection of test cases. Here it is important to remember that the more code paths and components involved the better. Here it is important to make a distinction between greenfield projects, where test cases will likely be created by a stakeholder or a few of them with the most business knowledge, or in legacy projects ( sometimes called “brownfield” although sometimes they deserve to be called “scorched earth” projects 😉 ), where it is not rare at all that source code has to be analyzed, to find undesirable coupling points, main code paths, unwritten business rules, kludges, etc. This is rarely taken into account. However there’s always a dark area that the stakeholders creating the pure logic cannot contemplate, and that is all the intricacies of internal state. Any sufficiently complex piece of software that has been maintained and enhanced over the years by different people will have more than its fair share of things such as hidden fields, flags with obscure names and purposes controlling flow, rare branches for particular situations multiplying the code paths, etc. It’s relatively easy to automate the testing of a calculator, not so easy to for a complex business application with hundred of inputs, edge cases, business scenarios, different workflows, etc.

The thing does not stop at selecting the cases; you need to take care of creating some sort of system, document, artifact that specifies the tests and leaves room to store the results, to at least, document the state of the system at a given point in time with regards to the test case selection, how many of them were satisfactory and how many weren’t.

Somebody needs to take care of executing that test case selection. Do you have dedicated people for testing? do you have some degree of test automation? According to the risk elements introduced in point 1, this gets more complicated and time consuming accordingly. Test execution is the part that’s probably more amenable to automation, but also architectural decisions made here can make it much harder. Think of distributed architectures, microservices that act / react asynchronously, hard – potentially hard to automate end to end.

Test result analysis. If all goes well, then nice, if not, errors have to be analyzed, traced, the input that caused them captured in order to reproduce scenarios. And we can also question ourselves about the quality of our oracle here, if it’s available and how accurate and correct it is. If we are to deduce correctness of test results from the specification, it may become very similar to exegesis of ancient texts and scriptures, that is, tedious, time consuming, error prone, unclear, and the basis for endless discussion about the interpretation of requirements.

Debugging of error scenarios. Potentially very time consuming. Fault removal and regression testing. You do have unit tests at least, right? Test maintenance. Tests and documentation maintenance seem to be the lowest caste in the software world. It’s not that they are often slashed off to “gain” time, it’s that nobody wants to remember to do it. If not updated, their usefulness obviously vanishes quickly. Bringing up to date an obsolete test suite is another nasty effort.

get your umbrellas out for the impending shitstorm

In the case of big and hairy integration projects between different systems and components this is even more important. My experience is that in this sort of project the fallacy of perfect execution and the siren song of perfect planning contribute a lot to underestimating the time and effort to integrate components that have been developed separately and the testing of the subsequent assembled system. Additional factors such as geographically diverse teams and if different companies have been involved can only make it more complicated and time consuming.

Now, let’s travel mentally to that beautiful rare moment of calm at the very beginning of the project. This is not an cool and agile startup. We’re now in an old corporate behemoth. We do things differently here. So, we’re in this meeting and we’re guesstimating our deadlines, internal and otherwise and ballparking a release date that gets then set into stone. We all agree that 3 days for testing is enough. Yi-haaaaaa. And we said 3 days because after all, this is a project integrating SAP, with a BPM, with a web site, with a set of services, with whatever you want. (the use of 3 is just an example)

So we did it. We set up ourselves for a lot of pain when the time comes to assemble everything and test it. The old engineering mentality kicked in. We haven’t learned yet that software isn’t like that. Now you’re bound to a nerve-racking project end, the grand finale, as nobody thought the integration would be that hard, we had specifications and countless meetings after all, right? and that so much testing and refinement would be needed. The problem is that this was not the first or the second time we did that kind of project. So what is keeping us from learning not to underestimate the importance of testing, the value that it adds to the product we put out there? Why do we keep making this mistake repeatedly? The answers to those questions can probably be very varied, depending on organizations, but most often, it will be variations on these

• We have to deliver faster and faster every day
• but we don’t have the right means or practices (we don’t write tests or do tdd, we don’t have continuous delivery or only a poor man version)
• this is a web project, and in the web you can have things done in 5 minutes
• We have promised this deadline to our stakeholders / clients – and yes, we promised that date the first thing in the project, before we knew what we had to do, the complexity of the challenge and even before consulting our tech staff , and as we promised this date our asses are now on the line

Thus we persist in these base mistakes and we ignore testing at our own peril and keep suffering as a consequence.