Do Developers Use Static Application Security Testing (SAST) Tools Straight Out of the Box? : A large-scale Empirical Study

Bennett, Gareth and Hall, Tracy and Counsell, Steve and Winter, Emily and Shippey, Thomas (2024) Do Developers Use Static Application Security Testing (SAST) Tools Straight Out of the Box? : A large-scale Empirical Study. In: Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2024 :. International Symposium on Empirical Software Engineering and Measurement . ACM, ESP, pp. 454-460. ISBN 9798400710476

Abstract

Static application Security Testing (SAST) tools are an established means of detecting vulnerabilities early in development. Previous studies have reported low detection rates from SAST tools and recommend either combining SAST tools or configuring rule sets to detect more vulnerabilities. However, while previous work suggests that developers rarely combine or configure any of the Automatic Static Analysis Tools (ASATs) they use, it is currently unclear whether SAST tools are used directly "out of the box". To understand how developers use SAST tools, we performed a large-scale survey involving 1,263 developers. We pre-screened developers to establish their SAST use and found that only 20% (204/1,003) used SAST tools. Of those developers who did use SAST tools, we found a large number did not use multiple tools (59%), did not configure tools (54%) or did neither (40%). Our results suggest that more work is needed to help developers combine and configure tools, since doing so is likely to detect significantly more vulnerabilities.