A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularization

Grokking the delayed transition from memorization to generalization in neural networks remains poorly understood, in part because prior empirical studies confound the roles of architecture, optimization, and regularization. We present a controlled study that systematically disentangles these factors on modular addition (mod 97), with matched and carefully tuned training regimes across models. Our central finding is that grokking dynamics are not primarily determined by architecture, but by interactions between optimization stability and regularization. Specifically, we show: (1) $\mathbf{\text{depth}}\mathbf{\text{has}}\mathbf{\text{a}}\mathbf{\text{non-monotonic}}\mathbf{\text{effect}}$, with depth-4 MLPs consistently failing to grok while depth-8 residual networks recover generalization, demonstrating that depth requires architectural stabilization; (2) $\mathbf{\text{the}}\mathbf{\text{apparent}}\mathbf{\text{gap}}\mathbf{\text{between}}\mathbf{\text{Transformers}}\mathbf{\text{and}}\mathbf{\text{MLPs}}\mathbf{\text{largely}}\mathbf{\text{disappears}}$ (1.11$\times$ delay) under matched hyperparameters, indicating that previously reported differences are largely due to optimizer and regularization confounds; (3) $\mathbf{\text{activation}}\mathbf{\text{function}}\mathbf{\text{effects}}\mathbf{\text{are}}\mathbf{\text{regime-dependent}}$, with GELU up to 4.3$\times$ faster than ReLU only when regularization permits memorization; and (4) $\mathbf{\text{weight}}\mathbf{\text{decay}}\mathbf{\text{is}}\mathbf{\text{the}}\mathbf{\text{dominant}}\mathbf{\text{control}}\mathbf{\text{parameter}}$, exhibiting a narrow ``Goldilocks’’ regime in which grokking occurs, while too little or too much prevents generalization. Across 3–5 seeds per configuration, these results provide a unified empirical account of grokking as an interaction-driven phenomenon. Our findings challenge architecture-centric interpretations and clarify how optimization and regularization jointly govern delayed generalization.