XiaoQi Han, Ru Li, Xiaoli Li, Jiye Liang, Zifang Zhang, and Jeff Pan. 2024. InstructEd: Soft-Instruction Tuning for Model Editing with Hops. In Findings of the Association for Computational Linguistics ACL 2024, pages 14953–14968, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
Abstract:
The task of model editing becomes popular for
correcting inaccurate or outdated parametric
knowledge in Large Language Models (LLMs).
However, there are major limitations of state of
the art (SOTA) model editing methods, including
the excessive memorization issue caused
by the direct editing methods, as well as the
error propagation and knowledge conflict issues
from the memory enhancement methods,
resulting in hindering models’ portability, e.g.,
the ability to transfer the new knowledge to related
one-hop or multi-hop content. To address
these issues, we propose the InstructEd method,
the idea of which is to insert soft instructions
into the attention module so as to facilitate interactions
between instructions and questions
and to understand and utilize new facts. Our
main findings are: (i) InstructEd has achieved
SOTA performance on three datasets for onehop/
multi-hop evaluation with LLaMAs and
GPT2, achieving 10% (5%) improvement in
one-hop (multi-hop) model editing. (ii) Different
from earlier methods on editing parameters
in FFN, we show that editing attention
can also help. (iii) Model editing is highly related
to retrieval augmented methods, which
can help improve the locality of model editing
while slightly decrease the editing performance
with hops. Our code is available at
https://github.com/sev777/InstructED.
License type:
Attribution 4.0 International (CC BY 4.0)
Funding Info:
This work has been supported by the National Natural Science Foundation of China (No.61936012), by the Science and Technology Cooperation and Exchange Special Project of ShanXi Province
(No.202204041101016), by the Chang Jiang Scholars Program (J2019032), and by the Key Research and Development Program of Shanxi Province (No.202102020101008).