Kavli Affiliate: Xiang Zhang
| First 5 Authors: Mengsong Wu, Tong Zhu, Han Han, Chuanyuan Tan, Xiang Zhang
| Summary:
This paper presents a new tool learning dataset Seal-Tools, which contains
self-instruct API-like tools. Seal-Tools not only offers a large number of
tools, but also includes instances which demonstrate the practical application
of tools. Seeking to generate data on a large scale while ensuring reliability,
we propose a self-instruct method to generate tools and instances, allowing
precise control over the process. Moreover, our Seal-Tools contains hard
instances that call multiple tools to complete the job, among which some are
nested tool callings. For precise and comprehensive evaluation, we use strict
format control and design three metrics from different dimensions. Therefore,
Seal-Tools can serve as a new benchmark to evaluate the tool-calling ability of
LLMs. Finally, we evaluate several prevalent LLMs and our finetuned model on
Seal-Tools. The results show that current systems are far from perfect. The
code, data and experiment results are available at
https://github.com/fairyshine/Seal-Tools .
| Search Query: ArXiv Query: search_query=au:”Xiang Zhang”&id_list=&start=0&max_results=3