Density of Neural Network Classes on Compact Subsets of Topological Vector Spaces

We prove density results for neural-network classes on compact sets $K\subset X$, where $X$ is a topological vector space whose continuous dual $X^{\ast }$ separates points. Let $\mathrm{\Psi }:\mathbb{R}\to \mathbb{R}$ be a continuous squashing function. We show that the class $${\mathrm{\Sigma }}_X(\mathrm{\Psi })=\{\sum _{j=1}^N{\omega }_j\mathrm{\Psi }(f_j(x)+b_j):N\in \mathbb{N},{\omega }_j,b_j\in \mathbb{R},f_j\in X^{\ast }\}$$ is dense in $C(K)$ with respect to the uniform norm. As a consequence, if $\mu$ is a Radon probability measure supported on $K$, then ${\mathrm{\Sigma }}_X(\mathrm{\Psi })$ is dense in $L^p(K,\mu )$ for every $1\le p<\mathrm{\infty }$.