Lee, G. C. F., Khu, D., Guretno, F., & Kurniawan, E. (2024). Specializing Language Models for 3GPP Standards: Enhancements for Technical Document Queries. 2024 IEEE Globecom Workshops (GC Wkshps), 1–6. https://doi.org/10.1109/gcwkshp64532.2024.11100864
Abstract:
This paper presents a novel approach to enhancing open-source language models for querying Third Generation Partnership Project (3GPP)-related technical documents, utilizing multiple-choice questions from the TeleQnA dataset as part of an International Telecommunication Union (ITU) Artificial Intelligence/Machine Learning (AI/ML) in 5G Challenge.1 Our primary focus is on the Phi-2 model, demonstrating that the integration of appropriately designed Retrieval-Augmented Generation (RAG), prompt engineering, and fine-tuning significantly enhances performance in handling complex technical standards-related queries. Our methodology leverages natural language processing techniques and re-ranking strategies, optimization of prompt ordering, and model fine-tuning. With our proposed methodology, we achieve an accuracy of 79.65% on a held-out test set based on TeleQnA. We address the challenges associated with adapting small language models to domain-specific tasks, offering insights into effective techniques for improving model performance within a resource-constrained setting. This research contributes to the field of telecommunications and language modeling, offering practical implications for future research and applications in this domain.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the Infocomm Media Development Authority - Future Communications Research & Development Programme
Grant Reference no. : FCP-NTU-RG-2022-021