#Bert finetune mods

Yet explaining their decisions is difficult despite recent work probing their internal representations.

Publisher = 'Association for Computational Linguistics',Ībstract = 'Large pretrained language models like BERT, after fine-tuning to a downstream task, have achieved high performance on a variety of NLP problems.

#Bert finetune mods

Cite (Informal): How does BERT’s attention change when you fine-tune? An analysis methodology and a case study in negation scope (Zhao & Bethard, ACL 2020) Copy Citation: BibTeX Markdown MODS XML Endnote More options… PDF: Video: = 'How does s attention change when you fine-tune? An analysis methodology and a case study in negation scope',īooktitle = 'Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics', Association for Computational Linguistics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4729–4747, Online. How does BERT’s attention change when you fine-tune? An analysis methodology and a case study in negation scope. Anthology ID: 2020.acl-main.429 Volume: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics Month: July Year: 2020 Address: Online Venue: ACL SIG: Publisher: Association for Computational Linguistics Note: Pages: 4729–4747 Language: URL: DOI: 10.18653/v1/2020.acl-main.429 Bibkey: zhao-bethard-2020-berts Cite (ACL): Yiyun Zhao and Steven Bethard. However, only the base models (not the large models) improve compared to a control task, indicating there is evidence for a shallow encoding of negation only in the base models. We find that after fine-tuning BERT and RoBERTa on a negation scope task, the average attention head improves its sensitivity to negation and its attention consistency across negation datasets compared to the pre-trained models. We apply this methodology to test BERT and RoBERTa on a hypothesis that some attention heads will consistently attend from a word in negation scope to the negation cue.

We propose a procedure and analysis methods that take a hypothesis of how a transformer-based model might encode a linguistic phenomenon, and test the validity of that hypothesis based on a comparison between knowledge-related downstream tasks with downstream control tasks, and measurement of cross-dataset consistency. Abstract Large pretrained language models like BERT, after fine-tuning to a downstream task, have achieved high performance on a variety of NLP problems.