Generative AI for corpus approaches to discourse studies : A critical evaluation of ChatGPT

Curry, Niall and Baker, Paul and Brookes, Gavin (2024) Generative AI for corpus approaches to discourse studies : A critical evaluation of ChatGPT. Applied Corpus Linguistics, 4 (1): 100082. ISSN 2666-7991

Text (ACL_ChatGPT_V3) - Accepted Version
Restricted to Repository staff only until 1 January 2040.
Available under License Creative Commons Attribution.
Download (0B)

Text (ACL_ChatGPT_V3)
ACL_ChatGPT_V3.pdf - Accepted Version
Available under License Creative Commons Attribution.
Download (421kB)

Abstract

This paper explores the potential of generative artificial intelligence technology, specifically ChatGPT, for advancing corpus approaches to discourse studies. The contribution of artificial intelligence technologies to linguistics research has been transformational, both in the contexts of corpus linguistics and discourse analysis. However, shortcomings in the efficacy of such technologies for conducting automated qualitative analysis have limited their utility for corpus approaches to discourse studies. Acknowledging that new technologies in data analysis can replace and supplement existing approaches, and in view of the potential affordances of ChatGPT for automated qualitative analysis, this paper presents three replication case studies designed to investigate the applicability of ChatGPT for supporting automated qualitative analysis within studies using corpus approaches to discourse analysis. The findings indicate that, generally, ChatGPT performs reasonably well when semantically categorising keywords; however, as the categorisation is based on decontextualised keywords, the categories can appear quite generic, limiting the value of such an approach for analysing corpora representing specialised genres and/or contexts. For concordance analysis, ChatGPT performs poorly, as the results include false inferences about the concordance lines and, at times, modifications of the input data. Finally, for function-to-form analysis, ChatGPT also performs poorly, as it fails to identify and analyse direct and indirect questions. Overall, the results raise questions about the affordances of ChatGPT for supporting automated qualitative analysis within corpus approaches to discourse studies, signalling issues of repeatability and replicability, ethical challenges surrounding data integrity, and the challenges associated with using non-deterministic technology for empirical linguistic research.