Zhang, Y., Geng, X., Liao, L., Sun, J., Yu, M., & Yu, G. (2025). MeMoTune: A Measure and Moment-Driven Fine-Tuning Framework for Quantized Large Language Models. Findings of the Association for Computational Linguistics: ACL 2025, 4036–4050. https://doi.org/10.18653/v1/2025.findings-acl.208
Abstract:
Quantizing large language models (LLMs) is essential for reducing memory and computational costs in natural language processing. Existing methods combine quantization with parameter-efficient fine-tuning but often fail to meet practical performance requirements. This paper introduces MeMoTune, a novel fine-tuning framework for quantized LLMs. By employing a measure and moment approach within a low-rank approximation framework in probability measure space, MeMoTune optimizes the objective function for superior fine-tuning results. The update process is further refined through scaled gradient, enhancing convergence efficiency and noise robustness. Experiments on tasks like text generation, summarization, and understanding show MeMoTune significantly outperforms state-of-the-art methods, e.g. fine-tuning Llama2-13B on GSM8K improves accuracy by 5.5%, while fine-tuning DeBERTaV3-base on CoLA of GLUE increases Matthews correlation by 1.7%.
License type:
Attribution 4.0 International (CC BY 4.0)
Funding Info:
This research / project is supported by the Agency for Science, Technology and Research - Manufacturing, Trade, and Connectivity Programmatic Fund: Towards Realistic Deep Learning for 3D Vision
Grant Reference no. : M23L7b0021
Description:
It was published in the Findings of the Association for Computational Linguistics (ACL 2025)