Kavli Affiliate: Ke Wang
| First 5 Authors: Ruixuan Xiao, Wentao Ma, Ke Wang, Yuchuan Wu, Junbo Zhao
| Summary:
LLM-based agents have emerged as promising tools, which are crafted to
fulfill complex tasks by iterative planning and action. However, these agents
are susceptible to undesired planning hallucinations when lacking specific
knowledge for expertise-intensive tasks. To address this, preliminary attempts
are made to enhance planning reliability by incorporating external
workflow-related knowledge. Despite the promise, such infused knowledge is
mostly disorganized and diverse in formats, lacking rigorous formalization and
comprehensive comparisons. Motivated by this, we formalize different formats of
workflow knowledge and present FlowBench, the first benchmark for
workflow-guided planning. FlowBench covers 51 different scenarios from 6
domains, with knowledge presented in diverse formats. To assess different LLMs
on FlowBench, we design a multi-tiered evaluation framework. We evaluate the
efficacy of workflow knowledge across multiple formats, and the results
indicate that current LLM agents need considerable improvements for
satisfactory planning. We hope that our challenging benchmark can pave the way
for future agent planning research.
| Search Query: ArXiv Query: search_query=au:”Ke Wang”&id_list=&start=0&max_results=3