Kavli Affiliate: Peter Ford
| First 5 Authors: Grgur Kovač, Rémy Portelas, Masataka Sawayama, Peter Ford Dominey, Pierre-Yves Oudeyer
| Summary:
The standard way to study Large Language Models (LLMs) through benchmarks or
psychology questionnaires is to provide many different queries from similar
minimal contexts (e.g. multiple choice questions). However, due to LLM’s highly
context-dependent nature, conclusions from such minimal-context evaluations may
be little informative about the model’s behavior in deployment (where it will
be exposed to many new contexts). We argue that context-dependence should be
studied as another dimension of LLM comparison alongside others such as
cognitive abilities, knowledge, or model size. In this paper, we present a
case-study about the stability of value expression over different contexts
(simulated conversations on different topics), and as measured using a standard
psychology questionnaire (PVQ) and a behavioral downstream task. We consider 19
open-sourced LLMs from five families. Reusing methods from psychology, we study
Rank-order stability on the population (interpersonal) level, and Ipsative
stability on the individual (intrapersonal) level. We explore two settings:
with and without instructing LLMs to simulate particular personalities. We
observe similar trends in the stability of models and model families – Mixtral,
Mistral and Qwen families being more stable than LLaMa-2 and Phi – over those
two settings, two different simulated populations, and even in the downstream
behavioral task. When instructed to simulate particular personas, LLMs exhibit
low Rank-Order stability, and this stability further diminishes with
conversation length. This highlights the need for future research directions on
LLMs that can coherently simulate a diversity of personas, as well as how
context-dependence can be studied in more thorough and efficient ways. This
paper provides a foundational step in that direction, and, to our knowledge, it
is the first study of value stability in LLMs.
| Search Query: ArXiv Query: search_query=au:”Peter Ford”&id_list=&start=0&max_results=3