Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challenges

Existing benchmarks that assess Language Models (LMs) as Language Agents (LAs) for tool use primarily focus on stateless, single-turn interactions or partial evaluations, such as tool selection in a single turn, overlooking the inherent stateful nature of interactions in multi-turn applications. To fulfill this gap, we propose $\mathtt{\text{DialogTool}}$, a multi-turn dialogue dataset with stateful tool interactions considering the whole life cycle of tool use, across six key tasks in three stages: 1) $\mathit{\text{tool}}\mathit{\text{creation}}$; 2) $\mathit{\text{tool}}\mathit{\text{utilization}}$: tool awareness, tool selection, tool execution; and 3) $\mathit{\text{role-consistent}}\mathit{\text{response}}$: response generation and role play. Furthermore, we build $\mathtt{\text{VirtualMobile}}$ – an embodied virtual mobile evaluation environment to simulate API calls and assess the robustness of the created APIs\footnote{We will use tools and APIs alternatively, there are no significant differences between them in this paper.}. Taking advantage of these artifacts, we conduct comprehensive evaluation on 13 distinct open- and closed-source LLMs and provide detailed analysis at each stage, revealing that the existing state-of-the-art LLMs still cannot perform well to use tools over long horizons.