Eigenvalue distribution of the Neural Tangent Kernel in the quadratic scaling

We compute the asymptotic eigenvalue distribution of the neural tangent kernel of a two-layer neural network under a specific scaling of dimension. Namely, if $X\in {\mathbb{R}}^{n\times d}$ is an i.i.d random matrix, $W\in {\mathbb{R}}^{d\times p}$ is an i.i.d $\mathcal{N}(0,1)$ matrix and $D\in {\mathbb{R}}^{p\times p}$ is a diagonal matrix with i.i.d bounded entries, we consider the matrix $$\mathrm{N}\mathrm{T}\mathrm{K}=\frac{1}{d}X{X}^{\mathrm{\top }}\odot \frac{1}{p}{\sigma }^{\mathrm{\prime }}\left(\frac{1}{\sqrt{d}}XW\right)D^2{\sigma }^{\mathrm{\prime }}{\left(\frac{1}{\sqrt{d}}XW\right)}^{\mathrm{\top }}$$ where ${\sigma }^{\mathrm{\prime }}$ is a pseudo-Lipschitz function applied entrywise and under the scaling $\frac{n}{dp}\to {\gamma }_1$ and $\frac{p}{d}\to {\gamma }_2$. We describe the asymptotic distribution as the free multiplicative convolution of the Marchenko–Pastur distribution with a deterministic distribution depending on $\sigma$ and $D$.