Kavli Affiliate: Wei Gao
| First 5 Authors: Joseph J. P. Simons, Wong Liang Ze, Prasanta Bhattacharya, Brandon Siyuan Loh, Wei Gao
| Summary:
Large language models are increasingly being used to label or rate
psychological features in text data. This approach helps address one of the
limiting factors of digital trace data – their lack of an inherent target of
measurement. However, this approach is also a form of psychological measurement
(using observable variables to quantify a hypothetical latent construct). As
such, these ratings are subject to the same psychometric considerations of
reliability and validity as more standard psychological measures. Here we
present a workflow for developing and evaluating large language model based
measures of psychological features which incorporate these considerations. We
also provide an example, attempting to measure the previously established
constructs of attitude certainty, importance and moralization from text. Using
a pool of prompts adapted from existing measurement instruments, we find they
have good levels of internal consistency but only partially meet validity
criteria.
| Search Query: ArXiv Query: search_query=au:”Wei Gao”&id_list=&start=0&max_results=3