Kavli Affiliate: Yi Zhou
| First 5 Authors: Gaifan Zhang, Gaifan Zhang, , ,
| Summary:
Semantic similarity between two sentences depends on the aspects considered
between those sentences. To study this phenomenon, Deshpande et al. (2023)
proposed the Conditional Semantic Textual Similarity (C-STS) task and annotated
a human-rated similarity dataset containing pairs of sentences compared under
two different conditions. However, Tu et al. (2024) found various annotation
issues in this dataset and showed that manually re-annotating a small portion
of it leads to more accurate C-STS models. Despite these pioneering efforts,
the lack of large and accurately annotated C-STS datasets remains a blocker for
making progress on this task as evidenced by the subpar performance of the
C-STS models. To address this training data need, we resort to Large Language
Models (LLMs) to correct the condition statements and similarity ratings in the
original dataset proposed by Deshpande et al. (2023). Our proposed method is
able to re-annotate a large training dataset for the C-STS task with minimal
manual effort. Importantly, by training a supervised C-STS model on our cleaned
and re-annotated dataset, we achieve a 5.4% statistically significant
improvement in Spearman correlation. The re-annotated dataset is available at
https://LivNLP.github.io/CSTS-reannotation.
| Search Query: ArXiv Query: search_query=au:”Yi Zhou”&id_list=&start=0&max_results=3