تقرير
Continued Pretraining for Better Zero- and Few-Shot Promptability
العنوان: | Continued Pretraining for Better Zero- and Few-Shot Promptability |
---|---|
المؤلفون: | Wu, Zhaofeng, Logan IV, Robert L., Walsh, Pete, Bhagia, Akshita, Groeneveld, Dirk, Singh, Sameer, Beltagy, Iz |
سنة النشر: | 2022 |
المجموعة: | Computer Science |
مصطلحات موضوعية: | Computer Science - Computation and Language |
الوصف: | Recently introduced language model prompting methods can achieve high accuracy in zero- and few-shot settings while requiring few to no learned task-specific parameters. Nevertheless, these methods still often trail behind full model finetuning. In this work, we investigate if a dedicated continued pretraining stage could improve "promptability", i.e., zero-shot performance with natural language prompts or few-shot performance with prompt tuning. We reveal settings where existing continued pretraining methods lack promptability. We also identify current methodological gaps, which we fill with thorough large-scale experiments. We demonstrate that a simple recipe, continued pretraining that incorporates a trainable prompt during multi-task learning, leads to improved promptability in both zero- and few-shot settings compared to existing methods, up to 31% relative. On the other hand, we find that continued pretraining using MAML-style meta-learning, a method that directly optimizes few-shot promptability, yields subpar performance. We validate our findings with two prompt tuning methods, and, based on our results, we provide concrete recommendations to optimize promptability for different use cases. Comment: EMNLP 2022 |
نوع الوثيقة: | Working Paper |
URL الوصول: | http://arxiv.org/abs/2210.10258 |
رقم الأكسشن: | edsarx.2210.10258 |
قاعدة البيانات: | arXiv |
الوصف غير متاح. |