Small Language Model as Data Prospector for Large Language Model

The quality of instruction data directly affects the performance of fine-tuned Large Language Models (LLMs). Previously, \cite{li2023one} proposed $\mathtt{\text{NUGGETS}}$, which identifies and selects high-quality quality data from a large dataset by identifying those individual instruction examples that can significantly improve the performance of different tasks after being learnt as one-shot instances. In this work, we propose $\mathtt{\text{SuperNUGGETS}}$, an improved variant of $\mathtt{\text{NUGGETS}}$ optimised for efficiency and performance. Our $\mathtt{\text{SuperNUGGETS}}$ uses a small language model (SLM) instead of a large language model (LLM) to filter the data for outstanding one-shot instances and refines the predefined set of tests. The experimental results show that the performance of $\mathtt{\text{SuperNUGGETS}}$ only decreases by 1-2% compared to $\mathtt{\text{NUGGETS}}$, but the efficiency can be increased by a factor of 58. Compared to the original $\mathtt{\text{NUGGETS}}$, our $\mathtt{\text{SuperNUGGETS}}$ has a higher utility value due to the significantly lower resource consumption.