Modularity of $d$-elliptic loci with level structure

We consider the generating series of special cycles on ${\mathcal{A}}_1(N)\times {\mathcal{A}}_g(N)$, with full level $N$ structure, valued in the cohomology of degree $2g$. The modularity theorem of Kudla-Millson for locally symmetric spaces implies that these series are modular. When $N=1$, the images of these loci in ${\mathcal{A}}_g$ are the $d$-elliptic Noether-Lefschetz loci, which are conjectured to be modular. In the appendix, it is shown that the resulting modular forms are nonzero for $g=2$ when $N\ge 11$ and $N\mathrm{\ne }12$.