Kavli Affiliate: Ting Xu
| First 5 Authors: Jie Zhang, Jie Zhang, , ,
| Summary:
Writing is a universal cultural technology that reuses vision for symbolic
communication. Humans display striking resilience: we readily recognize words
even when characters are fragmented, fused, or partially occluded. This paper
investigates whether advanced vision language models (VLMs) share this
resilience. We construct two psychophysics inspired benchmarks across distinct
writing systems, Chinese logographs and English alphabetic words, by splicing,
recombining, and overlaying glyphs to yield ”visible but unreadable” stimuli
for models while remaining legible to humans. Despite strong performance on
clean text, contemporary VLMs show a severe drop under these perturbations,
frequently producing unrelated or incoherent outputs. The pattern suggests a
structural limitation: models heavily leverage generic visual invariances but
under rely on compositional priors needed for robust literacy. We release
stimuli generation code, prompts, and evaluation protocols to facilitate
transparent replication and follow up work. Our findings motivate architectures
and training strategies that encode symbol segmentation, composition, and
binding across scripts, and they delineate concrete challenges for deploying
multimodal systems in education, accessibility, cultural heritage, and
security.
| Search Query: ArXiv Query: search_query=au:”Ting Xu”&id_list=&start=0&max_results=3