Kavli Affiliate: Max Tegmark
| First 5 Authors: Xiaoqing Sun, Alessandro Stolfo, Joshua Engels, Ben Wu, Senthooran Rajamanoharan
| Summary:
Sparse autoencoders (SAEs) are designed to extract interpretable features
from language models by enforcing a sparsity constraint. Ideally, training an
SAE would yield latents that are both sparse and semantically meaningful.
However, many SAE latents activate frequently (i.e., are emph{dense}), raising
concerns that they may be undesirable artifacts of the training procedure. In
this work, we systematically investigate the geometry, function, and origin of
dense latents and show that they are not only persistent but often reflect
meaningful model representations. We first demonstrate that dense latents tend
to form antipodal pairs that reconstruct specific directions in the residual
stream, and that ablating their subspace suppresses the emergence of new dense
features in retrained SAEs — suggesting that high density features are an
intrinsic property of the residual space. We then introduce a taxonomy of dense
latents, identifying classes tied to position tracking, context binding,
entropy regulation, letter-specific output signals, part-of-speech, and
principal component reconstruction. Finally, we analyze how these features
evolve across layers, revealing a shift from structural features in early
layers, to semantic features in mid layers, and finally to output-oriented
signals in the last layers of the model. Our findings indicate that dense
latents serve functional roles in language model computation and should not be
dismissed as training noise.
| Search Query: ArXiv Query: search_query=au:”Max Tegmark”&id_list=&start=0&max_results=3