Auto-tuning MPI Collective Operations on Large-Scale Parallel Systems

Zheng, Wenxu and Fang, Jianbin and Chen, Juan and Pan, Xiaodong and Wang, Hao and Huang, Chun and Sun, Xiaole and Tang, Tao and Wang, Zheng (2019) Auto-tuning MPI Collective Operations on Large-Scale Parallel Systems. In: The 21st IEEE International Conference on High Performance Computing and Communications :. IEEE, pp. 670-677. ISBN 9781728120591

Text (1570538287)
1570538287.pdf - Accepted Version
Available under License Creative Commons Attribution-NonCommercial.
Download (665kB)

Abstract

MPI libraries are widely used in applications of high performance computing. Yet, effective tuning of MPI colletives on large parallel systems is an outstanding challenge. This process often follows a trial-and-error approach and requires expert insights into the subtle interactions between software and the underlying hardware. This paper presents an empirical approach to choose and switch MPI communication algorithms at runtime to optimize the application performance. We achieve this by first modeling offline, through microbenchmarks, to find how the runtime parameters with different message sizes affect the choice of MPI communication algorithms. We then apply the knowledge to automatically optimize new unseen MPI programs. We evaluate our approach by applying it to NPB and HPCC benchmarks on a 384-node computer cluster of the Tianhe-2 supercomputer. Experimental results show that our approach achieves, on average, 22.7% (up to 40.7%) improvement over the default setting.