Neural Empirical Bayes

We unify $\mathit{\text{kernel}}\mathit{\text{density}}\mathit{\text{estimation}}$ and $\mathit{\text{empirical}}\mathit{\text{Bayes}}$ and address a set of problems in unsupervised learning with a geometric interpretation of those methods, rooted in the $\textit{concentration of measure}$phenomenon\mathrm{.}Kerneldensityisviewedsymbolicallyas$X\rightharpoonup Y$wheretherandomvariable$X$issmoothedto$Y= X+N(0,\sigma^2 I_d)$, and empirical Bayes is the machinery to denoise in a least-squares sense, which we express as $X\leftharpoondown Y$. A learning objective is derived by combining these two, symbolically captured by $X \rightleftharpoons Y$. Crucially, instead of using the original nonparametric estimators, we parametrize $\mathit{\text{the}}\mathit{\text{energy}}\mathit{\text{function}}$ with a neural network denoted by $\varphi$; at optimality, $\mathrm{\nabla }\varphi \approx -\mathrm{\nabla }\log f$ where $f$ is the density of $Y$. The optimization problem is abstracted as interactions of high-dimensional spheres which emerge due to the concentration of isotropic gaussians. We introduce two algorithmic frameworks based on this machinery: (i) a “walk-jump” sampling scheme that combines Langevin MCMC (walks) and empirical Bayes (jumps), and (ii) a probabilistic framework for $\mathit{\text{associative}}\mathit{\text{memory}}$, called NEBULA, defined `{a} la Hopfield by the $\mathit{\text{gradient}}\mathit{\text{flow}}$ of the learned energy to a set of attractors. We finish the paper by reporting the emergence of very rich “creative memories” as attractors of NEBULA for highly-overlapping spheres.