Self-DC: When to Reason and When to Act? Self Divide-and-Conquer for Compositional Unknown Questions

Previous research has typically concentrated on leveraging the internal knowledge of Large Language Models (LLMs) to answer known questions (i.e., $\mathit{\text{internal}}\mathit{\text{reasoning}}\mathit{\text{such}}\mathit{\text{as}}\mathit{\text{generate-then-read}}$). In contrast, for questions that fall outside their known scope, these models rely on external knowledge retrieval to provide accurate responses (i.e., $\mathit{\text{external}}\mathit{\text{acting}}\mathit{\text{such}}\mathit{\text{as}}\mathit{\text{retrieve-then-read}}$). However, few previous works consider the $\mathit{\text{compositional}}\mathit{\text{questions}}$, which consist of several known and unknown sub-questions, necessitating the dynamic combination of previous two methods (i.e., $\mathit{\text{internal}}\mathit{\text{reasoning}}\mathit{\text{and}}\mathit{\text{external}}\mathit{\text{acting}}$) to achieve a better trade-off between effectiveness and efficiency. To this end, we introduce a $\mathbf{\text{Self}}$ $\mathbf{\text{D}}$ivide-and-$\mathbf{\text{C}}$onquer (\textit{\texttt{Self-DC}}) framework, accompanying with the first $\mathbf{\text{C}}$ompositional $\mathbf{\text{u}}$nknown $\mathbf{\text{Q}}$uestion-$\mathbf{\text{A}}$nswering dataset (CuQA). This framework enables LLMs to adaptively choose between using internal knowledge and retrieving external knowledge as needed, resulting in a better trade-off between effectiveness and efficiency. Experimental results on two datasets demonstrate that \textit{\texttt{Self-DC}} can achieve comparable or even better performance with much fewer external calls compared with several strong baselines.