How Fine-Tuning Objectives Shape Layer-Wise Information in LLM Hallucination Detection

Under matched conditions, classifier-head fine-tuning ($\text{textsc}cls$) paradigm consistently outperforms the LLM-as-Judge sequence-to-sequence ($\text{textsc}s2s$) on binary hallucination detection. We trace this gap to the representations induced by the two objectives. Static analysis shows that $\text{textsc}cls$ learns a task-specific discriminative direction, pushing hidden states into a clear two-cluster geometry with stronger class separation and linearly decodable label information. By contrast, $\text{textsc}s2s$ relies on the pretrained language-modeling head to make decisions. This minimizes the pressure to reorganize representations, leaving the hidden states much closer to the pretrained backbone. Our analysis also reveals that the $\text{textsc}cls$ signal saturates before the final layer, after which upper layers form a plateau of statistically interchangeable and largely redundant representations. However, endpoint redundancy alone does not show whether these layers are dispensable during training. Further training-dynamics analysis reveals 1) a lower pretrained substrate, 2) a narrow upper-middle $\mathbf{\text{InfoWindow}}$ where task information emerges, and 3) a later-synchronizing upper plateau that mirrors this signal. This mechanism motivates us to remove layers above the InfoWindow. Truncating 31% of the network leaves accuracy unchanged within seed noise on Qwen3-4B and 8B and cuts latency by $1.4\times$. The same depth fraction, fixed before any analysis of the target models, transfers to Llama-3.2-3B and Mistral-7B-v0.3, and the layer structure behind it reappears there and on QNLI and SST-2. We hope this helps guide efficient task-specific LLM design.