Linear Independence of Generalized Neurons and Related Functions

The linear independence of neurons plays a significant role in theoretical analysis of neural networks. Specifically, given neurons $H_1,\mathrm{.}\mathrm{.}\mathrm{.},H_n:{\text{bR}}^N\times {\text{bR}}^d\to \text{bR}$, we are interested in the following question: when are $\{H_1({\theta }_1,\cdot ),\mathrm{.}\mathrm{.}\mathrm{.},H_n({\theta }_n,\cdot )\}$ are linearly independent as the parameters ${\theta }_1,\mathrm{.}\mathrm{.}\mathrm{.},{\theta }_n$ of these functions vary over ${\text{bR}}^N$. Previous works give a complete characterization of two-layer neurons without bias, for generic smooth activation functions. In this paper, we study the problem for neurons with arbitrary layers and widths, giving a simple but complete characterization for generic analytic activation functions.