Exploring the Problems, their Causes and Solutions of AI Pair Programming : A Study on Git Huband Stack Overflow

Zhou, Xiyu and Liang, Peng and Zhang, Beiqi and Li, Zengyang and Ahmad, Aakash and Shahin, Mojtaba and Waseem, Muhammad (2025) Exploring the Problems, their Causes and Solutions of AI Pair Programming : A Study on Git Huband Stack Overflow. Journal of Systems and Software, 219: 112204. ISSN 0164-1212

[thumbnail of JSS_accepted]
Text (JSS_accepted)
JSS_accepted.pdf - Accepted Version
Available under License Creative Commons Attribution.

Download (1MB)

Abstract

With the recent advancement of Artificial Intelligence (AI) and Large Language Models (LLMs), AI-based code generation tools become a practical solution for software development. GitHub Copilot, the AI pair programmer, utilizes machine learning models trained on a large corpus of code snippets to generate code suggestions using natural language processing. Despite its popularity in software development, there is limited empirical evidence on the actual experiences of practitioners who work with Copilot. To this end, we conducted an empirical study to understand the problems that practitioners face when using Copilot, as well as their underlying causes and potential solutions. We collected data from 473 GitHub issues, 706 GitHub discussions, and 142 Stack Overflow posts. Our results reveal that (1) Operation Issue and Compatibility Issue are the most common problems faced by Copilot users, (2) Copilot Internal Error, Network Connection Error, and Editor/IDE Compatibility Issue are identified as the most frequent causes, and (3) Bug Fixed by Copilot, Modify Configuration/Setting, and Use Suitable Version are the predominant solutions. Based on the results, we discuss the potential areas of Copilot for enhancement, and provide the implications for the Copilot users, the Copilot team, and researchers.

Item Type:
Journal Article
Journal or Publication Title:
Journal of Systems and Software
Uncontrolled Keywords:
/dk/atira/pure/subjectarea/asjc/1700/1708
Subjects:
?? hardware and architecturesoftwareinformation systems ??
ID Code:
224395
Deposited By:
Deposited On:
26 Sep 2024 14:10
Refereed?:
Yes
Published?:
Published
Last Modified:
26 Sep 2024 14:10