Delta-Audit: Explaining What Changes When Models Change

Model updates (new hyperparameters, kernels, depths, solvers, or data) change performance, but the \emph{reason} often remains opaque. We introduce $\mathbf{\text{Delta-Attribution}}$ (\mbox{$\mathrm{\Delta }$-Attribution}), a model-agnostic framework that explains \emph{what changed} between versions $A$ and $B$ by differencing per-feature attributions: $\mathrm{\Delta }\phi (x)={\phi }_B(x)-{\phi }_A(x)$. We evaluate $\mathrm{\Delta }\phi$ with a \emph{$\mathrm{\Delta }$-Attribution Quality Suite} covering magnitude/sparsity (L1, Top-$k$, entropy), agreement/shift (rank-overlap@10, Jensen–Shannon divergence), behavioural alignment (Delta Conservation Error, DCE; Behaviour–Attribution Coupling, BAC; CO$\mathrm{\Delta }$F), and robustness (noise, baseline sensitivity, grouped occlusion). Instantiated via fast occlusion/clamping in standardized space with a class-anchored margin and baseline averaging, we audit 45 settings: five classical families (Logistic Regression, SVC, Random Forests, Gradient Boosting, $k$NN), three datasets (Breast Cancer, Wine, Digits), and three A/B pairs per family. $\mathbf{\text{Findings.}}$ Inductive-bias changes yield large, behaviour-aligned deltas (e.g., SVC poly$\text{\hspace{0.17em}⁣}\to$rbf on Breast Cancer: BAC$\approx$0.998, DCE$\approx$6.6; Random Forest feature-rule swap on Digits: BAC$\approx$0.997, DCE$\approx$7.5), while ``cosmetic’’ tweaks (SVC $\mathtt{\text{gamma=scale}}$ vs.\ $\mathtt{\text{auto}}$, $k$NN search) show rank-overlap@10$=1.0$ and DCE$\approx$0. The largest redistribution appears for deeper GB on Breast Cancer (JSD$\approx$0.357). $\mathrm{\Delta }$-Attribution offers a lightweight update audit that complements accuracy by distinguishing benign changes from behaviourally meaningful or risky reliance shifts.