Task--Specificity Score: Measuring How Much Instructions Really Matter for Supervision

Instruction tuning is now the default way to train and adapt large language models, but many instruction–input–output pairs are only weakly specified: for a given input, the same output can remain plausible under several alternative instructions. This raises a simple question: \emph{does the instruction uniquely determine the target output?} We propose the $\mathbf{\text{Task–Specificity}}\mathbf{\text{Score}}\mathbf{\text{(TSS)}}$ to quantify how much an instruction matters for predicting its output, by contrasting the true instruction against plausible alternatives for the same input. We further introduce $\mathbf{\text{TSS++}}$, which uses hard alternatives and a small quality term to mitigate easy-negative effects. Across three instruction datasets ($\text{textsc}Alpaca$, $\text{textsc}Dolly-15k$, $\text{textsc}NI-20$) and three open LLMs (Gemma, Llama, Qwen), we show that selecting task-specific examples improves downstream performance under tight token budgets and complements quality-based filters such as perplexity and IFD.