Kavli Affiliate: Ke Wang
| First 5 Authors: Ke Wang, Tianyu Xia, Zhangxuan Gu, Yi Zhao, Shuheng Shen
| Summary:
Online GUI navigation on mobile devices has driven a lot of attention recent
years since it contributes to many real-world applications. With the rapid
development of large language models (LLM), multimodal large language models
(MLLM) have tremendous potential on this task. However, existing MLLMs need
high quality data to improve its abilities of making the correct navigation
decisions according to the human user inputs. In this paper, we developed a
novel and highly valuable dataset, named textbf{E-ANT}, as the first Chinese
GUI navigation dataset that contains real human behaviour and high quality
screenshots with annotations, containing nearly 40,000 real human traces over
5000+ different tinyAPPs. Furthermore, we evaluate various powerful MLLMs on
E-ANT and show their experiments results with sufficient ablations. We believe
that our proposed dataset will be beneficial for both the evaluation and
development of GUI navigation and LLM/MLLM decision-making capabilities.
| Search Query: ArXiv Query: search_query=au:”Ke Wang”&id_list=&start=0&max_results=3