Semgrep∗ : Improving the Limited Performance of Static Application Security Testing (SAST) Tools

Bennett, Gareth and Hall, Tracy and Winter, Emily and Counsell, Steve (2024) Semgrep∗ : Improving the Limited Performance of Static Application Security Testing (SAST) Tools. In: EASE '24 : Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. ACM International Conference Proceeding Series . The Association for Computing Machinery, ITA, pp. 614-623. ISBN 9798400717017

Abstract

Vulnerabilities in code should be detected and patched quickly to reduce the time in which they can be exploited. There are many automated approaches to assist developers in detecting vulnerabilities, most notably Static Application Security Testing (SAST) tools. However, no single tool detects all vulnerabilities and so relying on any one tool may leave vulnerabilities dormant in code. In this study, we use a manually curated dataset to evaluate four SAST tools on production code with known vulnerabilities. Our results show that the vulnerability detection rates of individual tools range from 11.2% to 26.5%, but combining these four tools can detect 38.8% of vulnerabilities. We investigate why SAST tools are unable to detect 61.2% of vulnerabilities and identify missing vulnerable code patterns from tool rule sets. Based on our findings, we create new rules for Semgrep, a popular configurable SAST tool. Our newly configured Semgrep tool detects 44.7% of vulnerabilities, more than using a combination of tools, and a 181% improvement in Semgrep's detection rate.