Kavli Affiliate: Yi Zhou
| First 5 Authors: Xiaohang Tang, Yi Zhou, Danushka Bollegala, ,
| Summary:
Dynamic contextualised word embeddings represent the temporal semantic
variations of words. We propose a method for learning dynamic contextualised
word embeddings by time-adapting a pretrained Masked Language Model (MLM) using
time-sensitive templates. Given two snapshots $C_1$ and $C_2$ of a corpora
taken respectively at two distinct timestamps $T_1$ and $T_2$, we first propose
an unsupervised method to select (a) pivot terms related to both $C_1$ and
$C_2$, and (b) anchor terms that are associated with a specific pivot term in
each individual snapshot. We then generate prompts by filling manually compiled
templates using the extracted pivot and anchor terms. Moreover, we propose an
automatic method to learn time-sensitive templates from $C_1$ and $C_2$,
without requiring any human supervision. Next, we use the generated prompts to
adapt a pretrained MLM to $T_2$ by fine-tuning it on the prompts. Experimental
results show that our proposed method significantly reduces the perplexity of
test sentences selected from $T_2$, thereby outperforming the current
state-of-the-art dynamic contextualised word embedding methods.
| Search Query: ArXiv Query: search_query=au:”Yi Zhou”&id_list=&start=0&max_results=10