Kavli Affiliate: Wei Gao
| First 5 Authors: Joseph J. P. Simons, Wong Liang Ze, Prasanta Bhattacharya, Brandon Siyuan Loh, Wei Gao
| Summary:
Digital trace data provide potentially valuable resources for understanding
human behaviour, but their value has been limited by issues of unclear
measurement. The growth of large language models provides an opportunity to
address this limitation in the case of text data. Specifically, recognizing
cases where their responses are a form of psychological measurement (the use of
observable indicators to assess an underlying construct) allows existing
measures and accuracy assessment frameworks from psychology to be re-purposed
to use with large language models. Based on this, we offer four methodological
recommendations for using these models to quantify text features: (1) identify
the target of measurement, (2) use multiple prompts, (3) assess internal
consistency, and (4) treat evaluation metrics (such as human annotations) as
expected correlates rather than direct ground-truth measures. Additionally, we
provide a workflow for implementing this approach.
| Search Query: ArXiv Query: search_query=au:”Wei Gao”&id_list=&start=0&max_results=3