Benchmarking Change Detection Exactness and Overhead of Instrumentation and Sampling

Reichelt, David Georg and Skarbalius, Juozas (2026) Benchmarking Change Detection Exactness and Overhead of Instrumentation and Sampling. In: ICPE Companion '26: Companion of the 17th ACM/SPEC International Conference on Performance Engineering :. ACM, New York. ISBN 9798400723261

Abstract

Observability of software systems requires the collection of runtime data. The collection can be performed using either sampling or instrumentation. These two techniques have different properties with respect to collectible data, the exactness of change detection, and the overhead in terms of resource consumption. During software development, these techniques are used to detect performance changes between different commits or variants of the developed software. While prior comparisons of these techniques focused on the accuracy of the measurements, we focus on the exactness of change detection. This paper presents: (1) A benchmark for comparing the change detection exactness and overhead of sampling and instrumentation, and (2) an experimental evaluation of this benchmark on current hardware. By our evaluation, we find the tracing overhead caused by instrumentation is 0.5 μ s per method call in our setting. At the same time, sampling does not cause overhead that can be identified with statistical significance. Also for change detection, sampling shows to be much more effective. Due to these measurements, we follow that developers should use instrumentation only if it is necessary to trace the full behavior of a system, e.g., if single REST requests need to be identified, or if it is possible to preselect a small amount of methods that should be observed. If no preselection of methods is possible and performance changes of complex programs should be obtained, sampling is the only feasible technique due to the high overhead of instrumentation.