Which Software Faults Are Tests Not Detecting?

Petric, J. and Hall, T. and Bowes, David (2020) Which Software Faults Are Tests Not Detecting? In: PROCEEDINGS of EASE 2020: Evaluation and Assessment in Software Engineering :. ACM, New York, pp. 160-169. ISBN 9781450377317

[thumbnail of Full paper]
Text (Full paper)
3383219.3383236.pdf - Accepted Version
Available under License Creative Commons Attribution-NonCommercial.

Download (177kB)


Context: Software testing plays an important role in assuring the reliability of systems. Assessing the efficacy of testing remains challenging with few established test effectiveness metrics. Those metrics that have been used (e.g. coverage and mutation analysis) have been criticised for insufficiently differentiating between the faults detected by tests. Objective: We investigate how effective tests are at detecting different types of faults and whether some types of fault evade tests more than others. Our aim is to suggest to developers specific ways in which their tests need to be improved to increase fault detection. Method: We investigate seven fault types and analyse how often each goes undetected in 10 open source systems. We statistically look for any relationship between the test set and faults. Results: Our results suggest that the fault detection rates of unit tests are relatively low, typically finding only about a half of all faults. In addition, conditional boundary and method call removals are less well detected by tests than other fault types. Conclusions: We conclude that the testing of these open source systems needs to be improved across the board. In addition, despite boundary cases being long known to attract faults, tests covering boundaries need particular improvement. Overall, we recommend that developers do not rely only on code coverage and mutation score to measure the effectiveness of their tests.

Item Type:
Contribution in Book/Report/Proceedings
Additional Information:
© ACM, 2020. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in PROCEEDINGS of EASE 2020: Evaluation and Assessment in Software Engineering http://doi.acm.org/10.1145/3383219.3383236
?? software testingtest effectivenessunit testsfault detectionopen source softwareopen systemssoftware reliabilitycode coveragefault detection ratefault typesmutation analysismutation scoreopen source systemsoftware faulttest effectivenesssoftware testing ??
ID Code:
Deposited By:
Deposited On:
08 Oct 2020 11:07
Last Modified:
16 Jul 2024 04:58