Kavli Affiliate: Jing Wang
| First 5 Authors: Xiaopeng Li, Shasha Li, Bin Ji, Shezheng Song, Xi Wang
| Summary:
Model editing has recently gained widespread attention. Current model editing
methods primarily involve modifying model parameters or adding additional
modules to the existing model. However, the former causes irreversible damage
to LLMs, while the latter incurs additional inference overhead and fuzzy vector
matching is not always reliable. To address these issues, we propose an
expandable Subject Word Embedding Altering (SWEA) framework, which modifies the
representation of subjects and achieve the goal of editing knowledge during the
inference stage. SWEA uses precise key matching outside the model and performs
reliable subject word embedding altering, thus protecting the original weights
of the model without increasing inference overhead. We then propose optimizing
then suppressing fusion method, which first optimizes the embedding vector for
the editing target and then suppresses the Knowledge Embedding Dimension (KED)
to obtain the final fused embedding. We thus propose SWEAOS method for editing
factual knowledge in LLMs. We demonstrate the state-of-the-art performance of
SWEAOS on the COUNTERFACT and zsRE datasets. To further validate the reasoning
ability of SWEAOS in editing knowledge, we evaluate it on the more complex
RIPPLEEDITS benchmark. The results on two subdatasets demonstrate that our
SWEAOS possesses state-of-the-art reasoning ability.
| Search Query: ArXiv Query: search_query=au:”Jing Wang”&id_list=&start=0&max_results=3