EXACT - Experimental Analysis of the Coupling Effect Hypothesis in Software Testing

In software testing, mutation analysis is a technique that systematically inserts simple bugs into a program under test. Once a set of buggy programs, known as mutants, has been created, they are run on a set of test cases. If all mutants fail, the set of test cases (or the technique selecting them) is deemed good. Fundamental to mutation analysis is the so-called coupling effect hypothesis. The hypothesis states that a set of test cases detecting most simple bugs in a program will also detect most complex faults in the same program. If the coupling effect does not exist for real software systems, then mutation analysis is not a reliable way of deciding whether or not a program is thoroughly tested. This would also cast doubt on the use of mutation analysis in research as a way of assessing the effectiveness of testing techniques.

The validity of any evidence on the coupling effect is limited by the extent to which the programs and faults studied are representative of real-world programs and naturally-occurring faults. However, nearly all existing studies of the coupling effect use small programs and artificial faults generated for the sake of the experiment. In this project, we will empirically study the coupling effect in large software systems with naturally-occurring faults. We expect to provide significant evidence supporting or refuting the hypothesis, allowing researchers and testers to determine whether mutation analysis is a meaningful way to assess their work.