EPFL Unveils MEMOIR: A Breakthrough Framework for Continuous Model Editing in Large Language Models

Challenges in Updating Large Language Models

Large Language Models (LLMs) excel in numerous tasks due to extensive pre-training on massive datasets. However, they often produce outdated or inaccurate information and may carry biases once deployed. Regular updates are essential to maintain their accuracy and reliability. Traditional fine-tuning approaches are costly and prone to catastrophic forgetting, where previously learned knowledge is lost. This challenge has driven interest in lifelong model editing, which aims to efficiently and locally update model knowledge. Effective edits must ensure reliability, generalizability, and locality. Non-parametric methods provide precise localized edits but lack in generalization, while parametric methods generalize better but are vulnerable to forgetting.

Limitations of Existing Model Editing Techniques

Previous research has explored sparse neural activations within continual learning frameworks. Approaches like PackNet and Supermasks-in-Superposition assign disjoint subsets of parameters to different tasks. Gradient-based methods such as GPM and SPARCL enhance efficiency through orthogonal updates but are mostly tailored for continual learning scenarios. Parametric techniques like ROME, MEMIT, and WISE apply weight modifications using locating-then-editing strategies or auxiliary modules but face forgetting issues over extended editing sequences. Non-parametric approaches, including GRACE and LOKA, preserve original weights by storing knowledge externally, allowing precise local edits; however, they depend on exact input matches and thus have limited generalization.

Introducing MEMOIR: Balancing Reliability, Generalization, and Locality

Researchers from EPFL have introduced MEMOIR (Model Editing with Minimal Overwrite and Informed Retention), designed to optimally balance reliability, generalization, and locality for large-scale model edits. MEMOIR incorporates a memory module composed of a fully-connected layer within a single transformer block where all edits are made. It combats catastrophic forgetting by assigning distinct parameter subsets to each edit and retrieving them during inference, activating only the relevant knowledge for specific prompts. The method leverages structured sparsification with sample-dependent masks during editing, activating only prompt-specific parameter subsets. This strategy distributes new information across the parameter space, minimizing overwriting and reducing forgetting.

Evaluation and Results

MEMOIR operates as a residual memory framework during inference, combining the original layer outputs with residual memory outputs. It was benchmarked against baselines including GRACE (external knowledge storage), DEFER (inference-time routing), causal tracing methods (ROME, MEMIT, ALPHAEDIT), memory-based methods (WISE), and direct fine-tuning. Tests were performed on four autoregressive language models: LLaMA-3-8B-Instruct, Mistral-7B, LLaMA-2-7B, and GPT-J-6B.

On the ZsRE question-answering dataset, MEMOIR achieved an average metric of 0.95 on LLaMA-3 with 1000 edits, outperforming previous methods by 0.16. Similar results were observed with Mistral, confirming the method's robustness. MEMOIR also maintained balanced performance with increasing edit volumes for hallucination correction using the SelfCheckGPT dataset. It preserved saturated locality scores even with 600 edits and achieved perplexity metrics 57% and 77% lower than WISE on LLaMA-3 and Mistral, respectively.

Future Directions

MEMOIR offers a scalable framework that successfully balances key editing criteria through innovative sparsification and selective activation. However, it currently edits only single linear layers, which may limit handling of long-term edits or knowledge requiring broader modifications. Future work includes extending the method to multiple layers, developing hierarchical editing strategies, and applying it to multi-modal or encoder-decoder models beyond decoder-only transformers.