Differentiable Window for Dynamic Local Attention

Page view(s)
20
Checked on Sep 21, 2022
Differentiable Window for Dynamic Local Attention
Title:
Differentiable Window for Dynamic Local Attention
Other Titles:
ACL 2020
DOI:
Publication URL:
Keywords:
Publication Date:
01 July 2020
Citation:
Abstract:
We propose Differentiable Window, a new neural module and general purpose component for dynamic window selection. While universally applicable, we demonstrate a compelling use case of utilizing Differentiable Window to improve standard attention modules by enabling more focused attentions over the input regions. We propose two variants of Differentiable Window, and integrate them within the Transformer architecture in two novel ways. We evaluate our proposed approach on a myriad of NLP tasks, including machine translation, sentiment analysis, subject-verb agreement and language modeling. Our experimental results demonstrate consistent and sizable improvements across all tasks.
License type:
PublisherCopyrights
Funding Info:
Description:
ISBN:

Files uploaded:

Files uploaded: