Perverse coherent extensions on Calabi-Yau threefolds and representations of cohomological Hall algebras

For $Y\to X$ a toric Calabi-Yau threefold resolution and $M\in {\text{DD}}^b\text{Coh}(Y{)}^T$ satisfying some hypotheses, we define a stack $\text{mf}M(Y,M)$ parameterizing \emph{perverse coherent extensions} of $M$, iterated extensions of $M$ and the compactly supported perverse coherent sheaves of Bridgeland. We define framed variants $\text{mf}{M}^{\text{f}}(Y,M)$, prove that they are equivalent to stacks of representations of framed quivers with potential $(Q^{\text{f}},W^{\text{f}})$, and deduce natural monad presentations for these sheaves. Moreover, following Soibelman we prove that the homology $H_{\bullet }(\text{mf}{M}^{\text{f},\zeta }(Y,M),{\phi }_{W^{\text{f}}})$ of the space of $\zeta$-stable, $\text{f}$-framed perverse coherent extensions of $M$, with coefficients in the sheaf ${\phi }_{W^{\text{f}}}$ of vanishing cycles for $W^{\text{f}}$, is a representation of the Kontsevich-Soibelman cohomological Hall algebra of $Y$. For $M=\text{mc}{O}_Y[1]$, $\text{mf}{M}^{\text{f}}(Y,M)$ is the stack of perverse coherent systems of Nagao-Nakajima, so $\text{bb}{V}_Y^{\zeta }=H_{\bullet }(\text{mf}{M}^{\text{f},\zeta }(Y,M),{\phi }_{W^{\text{f}}})$ is the DT/PT series of $Y$ for $\zeta ={\zeta }_{\text{DT}\mathrm{/}\text{PT}}$ by Szendroi and \emph{loc. cit.}, and we conjecture that ${\text{V}}_Y^{{\zeta }_{\text{NCDT}}}$ is the vacuum module for the quiver Yangian of Li-Yamazaki. For $M=\text{mc}{O}_S[1]$ with $S\subset Y$ a divisor, $\text{mf}{M}^{\text{f}}(Y,M)$ provides a definition in algebraic geometry for Nekrasov’s spiked instanton variant of the ADHM construction, and analogous variants of the constructions of Kronheimer-Nakajima, Nakajima-Yoshioka, and Finkelberg-Rybnikov. We conjecture that $H_{\bullet }(\text{mf}{M}^{\text{f},\zeta }(Y,M),{\phi }_{W^{\text{f}}})$ is the vacuum module of the vertex algebra $\text{V}(Y,S)$ defined by the \mbox{authors} in a companion paper, generalizing the AGT conjecture to this setting. For $Y\to X=\{xy-z^m{w}^n\}$, this gives a geometric approach to the relationship between $W$-algebras and Yangians for affine ${\text{gl}}_{m\mathrm{\mid }n}$.