Can We Edit Factual Knowledge by In-Context Learning?

التفاصيل البيبلوغرافية
العنوان: Can We Edit Factual Knowledge by In-Context Learning?
المؤلفون: Zheng, Ce, Li, Lei, Dong, Qingxiu, Fan, Yuxuan, Wu, Zhiyong, Xu, Jingjing, Chang, Baobao
سنة النشر: 2023
المجموعة: Computer Science
مصطلحات موضوعية: Computer Science - Computation and Language
الوصف: Previous studies have shown that large language models (LLMs) like GPTs store massive factual knowledge in their parameters. However, the stored knowledge could be false or out-dated. Traditional knowledge editing methods refine LLMs via fine-tuning on texts containing specific knowledge. However, with the increasing scales of LLMs, these gradient-based approaches bring large computation costs. The trend of model-as-a-service also makes it impossible to modify knowledge in black-box LMs. Inspired by in-context learning (ICL), a new paradigm based on demonstration contexts without parameter updating, we explore whether ICL can edit factual knowledge. To answer this question, we give a comprehensive empirical study of ICL strategies. Experiments show that in-context knowledge editing (IKE), without any gradient and parameter updating, achieves a competitive success rate compared to gradient-based methods on GPT-J (6B) but with much fewer side effects, including less over-editing on similar but unrelated facts and less knowledge forgetting on previously stored knowledge. We also apply the method to larger LMs with tens or hundreds of parameters like OPT-175B, which shows the scalability of our method. The code is available at https://github.com/Zce1112zslx/IKE.
نوع الوثيقة: Working Paper
URL الوصول: http://arxiv.org/abs/2305.12740
رقم الأكسشن: edsarx.2305.12740
قاعدة البيانات: arXiv