A Data-driven Latent Semantic Analysis for Automatic Text Summarization using LDA Topic Modelling

Onah, Daniel F. O. and Pang, Elaine L. L. and El-Haj, Mahmoud (2023) A Data-driven Latent Semantic Analysis for Automatic Text Summarization using LDA Topic Modelling. In: 2022 IEEE International Conference on Big Data (Big Data). 2022 IEEE International Conference on Big Data (Big Data) . IEEE, pp. 2771-2780. ISBN 9781665480468

Text (Accepted_Version_BigD220)
Accepted_Version_BigD220.pdf - Accepted Version
Available under License Creative Commons Attribution.

Download (9MB)


With the advent and popularity of big data mining and huge text analysis in modern times, automated text summarization became prominent for extracting and retrieving important information from documents. This research investigates aspects of automatic text summarization from the perspectives of single and multiple documents. Summarization is a task of condensing huge text articles into short, summarized versions. The text is reduced in size for summarization purpose but preserving key vital information and retaining the meaning of the original document. This study presents the Latent Dirichlet Allocation (LDA) approach used to perform topic modelling from summarised medical science journal articles with topics related to genes and diseases. In this study, PyLDAvis web-based interactive visualization tool was used to visualise the selected topics. The visualisation provides an overarching view of the main topics while allowing and attributing deep meaning to the prevalence individual topic. This study presents a novel approach to summarization of single and multiple documents. The results suggest the terms ranked purely by considering their probability of the topic prevalence within the processed document using extractive summarization technique. PyLDAvis visualization describes the flexibility of exploring the terms of the topics’ association to the fitted LDA model. The topic modelling result shows prevalence within topics 1 and 2. This association reveals that there is similarity between the terms in topic 1 and 2 in this study. The efficacy of the LDA and the extractive summarization methods were measured using Latent Semantic Analysis (LSA) and Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metrics to evaluate the reliability and validity of the model.

Item Type:
Contribution in Book/Report/Proceedings
Additional Information:
©2023 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE. Requirements for research articles as indicated by UKRI Open Access Policy requirements for in-scope research articles: We follow ROUTE 1: Publish the research article open access in a journal or publishing platform which makes the Version of Record immediately open access via its website; The Version of Record must be free and unrestricted to view and download. It must have a Creative Commons Attribution (CC BY) licence, or other licence permitted by UKRI.
Uncontrolled Keywords:
Research Output Funding/no_not_funded
ID Code:
Deposited By:
Deposited On:
09 Feb 2023 10:10
Last Modified:
25 Mar 2023 00:37