Emergent Hierarchical Structure in Large Language Models: An Information-Theoretic Framework for Multi-Scale Representation

Why do language models from different architecture families respond so differently to the same perturbation? We argue that the answer is not scale, but \emph{how architecture shapes information compression}. Analyzing eight Transformer models (7B–70B parameters) from the Llama and Qwen families, we show that every model spontaneously develops discrete functional boundaries dividing its layers into Local, Intermediate, and Global processing segments – yet boundary locations and per-segment brittleness are determined overwhelmingly by architecture family rather than model size or training configuration. We formalize this regularity as the $\mathbf{\text{Multi-Scale}}\mathbf{\text{Probabilistic}}\mathbf{\text{Generation}}\mathbf{\text{Theory}}$ (MSPGT), which models an autoregressive Transformer as a Hierarchical Variational Information Bottleneck system and derives a tiered set of falsifiable predictions. Three predictions are strongly confirmed: all eight models exhibit two prominent phase-transition boundaries (P1.1); Llama boundary positions are stable across a $10\times$ parameter range ($\mathrm{C}\mathrm{V}=0.067$–$0.095$) while Qwen positions vary widely ($\mathrm{C}\mathrm{V}=0.465$–$0.726$), precisely matching our strong- and weak-dominance conditions; and cross-architecture local-segment brittleness spans $\mathbf{\text{three}}\mathbf{\text{orders}}\mathbf{\text{of}}\mathbf{\text{magnitude}}$ ($493\times$ ratio) – a gap that architecture family alone predicts and that dwarfs any within-family or scale-driven variation.