G. C. F. Lee, D. Khu, F. Guretno, and E. Kurniawan, "Specializing Language Models for 3GPP Standards: Enhancements for Technical Document Queries," IEEE Globecom 2024, In Proceedings, Cape Town, South Africa, December 2024.
Abstract:
This paper presents a novel approach to enhancing open-source language models for querying Third Generation Partnership Project (3GPP)-related technical document documents, utilizing multiple-choice questions from the TeleQnA dataset as part of an International Telecommunication Union (ITU) Artificial Intelligence/Machine Learning (AI/ML) in 5G Challenge. Our primary focus is on the Phi-2 model, demonstrating that the integration of appropriately designed Retrieval-Augmented Generation (RAG), prompt engineering, and finetuning significantly enhances performance in handling complex technical standards-related queries. Our methodology leverages natural language processing techniques and re-ranking strategies, optimization of prompt ordering, and model fine-tuning. With our proposed methodology, we achieved an accuracy of 79.65% on a held-out test set based on TeleQnA. We address the challenges associated with adapting small models to domain-specific tasks, offering insights into effective techniques for improving model performance within a resource-constrained setting. This research contributes to the field of telecommunications and language modelling, offering practical implications for future research and applications in this domain.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the National Research Foundation, Singapore and Infocomm Media Development Authority - Future Communications Research & Development Programme
Grant Reference no. : FCP-NTU-RG-2022-021