He, X., Geng, X., Zhang, T., Yu, M., Zhao, Y., Yang, X., Wu, M., & Yu, G. (2026). MeLoRA : Probability measures-based low-rank adaptation with Gaussian variational inference. Knowledge-Based Systems, 336, 115318. https://doi.org/10.1016/j.knosys.2026.115318
Abstract:
Full parameter fine-tuning of large language models (LLMs) imposes prohibitive computational burdens for edge deployment. Parameter-Efficient Fine-Tuning (PEFT) has emerged as a critical paradigm that optimizes only task-specific parameter subsets rather than the full weight matrix. Among widely used PEFT methods, Low+-Rank Adaptation (LoRA) and its variants have gained considerable popularity as they avoid additional inference costs. However, deterministic LoRA methods remain susceptible to data noise and catastrophic forgetting, creating a persistent accuracy gap with full fine-tuning. To address these limitations, we propose MeLoRA, a probabilistic framework that reformulates the fine-tuning objective using Gaussian variational inference. Building upon established Bayesian principles, our key innovation lies in applying this probabilistic formulation to the low-rank adapter context. MeLoRA treats adapter parameters as random variables governed by learnable Gaussian distributions, thereby shifting optimization from point estimates to the distribution’s mean and covariance. This approach naturally captures parameter uncertainty and acts as an adaptive regularizer, enhancing robustness. We further employ a memory-efficient, low-rank parameterization of the covariance matrix to maintain practical efficiency. Extensive experiments with multiple pre-trained models across Natural Language Understanding (NLU), Natural Language Generation (NLG), and commonsense reasoning tasks validate the effectiveness of MeLoRA. Results show MeLoRA achieves significant improvements over baseline methods, e.g., fine-tuning LLaMA2-7B on the commonsense reasoning benchmark represents a 0.51% relative improvement, while fine-tuning GPT-2 Medium on E2E increases ROUGE-L by 0.6%.
License type:
Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)
Funding Info:
This research / project is supported by the A*STAR - Manufacturing, Trade, and Connectivity Programmatic Fund
Grant Reference no. : M23L7b0021