Stopwords in technical language processing

Page view(s)
Checked on Jul 13, 2022
Stopwords in technical language processing
Stopwords in technical language processing
Other Titles:
Publication Date:
05 August 2021
Sarica, S., & Luo, J. (2021). Stopwords in technical language processing. PLOS ONE, 16(8), e0254937. doi:10.1371/journal.pone.0254937
There are increasing applications of natural language processing techniques for information retrieval, indexing, topic modelling and text classification in engineering contexts. A standard component of such tasks is the removal of stopwords, which are uninformative components of the data. While researchers use readily available stopwords lists that are derived from non-technical resources, the technical jargon of engineering fields contains their own highly frequent and uninformative words and there exists no standard stopwords list for technical language processing applications. Here we address this gap by rigorously identifying generic, insignificant, uninformative stopwords in engineering texts beyond the stopwords in general texts, based on the synthesis of alternative statistical measures such as term frequency, inverse document frequency, and entropy, and curating a stopwords dataset ready for technical language processing applications.
License type:
Attribution 4.0 International (CC BY 4.0)
Funding Info:
This research / project is supported by the A*STAR - K-EMERGE: Knowledge Extraction, Modelling, and Explainable Reasoning for General Expertise programme
Grant Reference no. : A19E2b0098