Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on H\"older Class

In this paper, we construct neural networks with ReLU, sine and $2^x$ as activation functions. For general continuous $f$ defined on $[0,1{]}^d$ with continuity modulus ${\omega }_f(\cdot )$, we construct ReLU-sine-$2^x$ networks that enjoy an approximation rate $\mathcal{O}({\omega }_f(\sqrt{d})\cdot 2^{-M}+{\omega }_f\left(\frac{\sqrt{d}}{N}\right))$, where $M,N\in {\mathbb{N}}^{+}$ denote the hyperparameters related to widths of the networks. As a consequence, we can construct ReLU-sine-$2^x$ network with the depth $5$ and width $\max\left{\left\lceil2d^{3/2}\left(\frac{3\mu}{\epsilon}\right)^{1/{\alpha}}\right\rceil,2\left\lceil\log_2\frac{3\mu d^{\alpha/2}}{2\epsilon}\right\rceil+2\right}$thatapproximates$f\in $\mathcal{H}${\mu}^{\alpha}([0,1]^d)$withinagiventolerance$\epsilon >0$measuredin$L^p$norm$p\in[1,\infty)$,where$$\mathcal{H}${\mu}^{\alpha}([0,1]^d)$ denotes the H"older continuous function class defined on $[0,1{]}^d$ with order $\alpha \in (0,1]$ and constant $\mu > 0$\mathrm{.}Therefore,theReLU-sine-$2^x$ networks overcome the curse of dimensionality on ${\mathcal{H}}_{\mu }^{\alpha }([0,1{]}^d)$. In addition to its supper expressive power, functions implemented by ReLU-sine-$2^x$ networks are (generalized) differentiable, enabling us to apply SGD to train.