Moduli spaces of framed symplectic and orthogonal bundles on P2 and the K-theoretic Nekrasov partition functions

Let $K$ be the compact Lie group $USp(N\mathrm{/}2)$ or $SO(N,R)$. Let $M_n^K$ be the moduli space of framed K-instantons over $S^4$ with the instanton number $n$. By Donaldson (1984), $M_n^K$ is endowed with a natural scheme structure. It is a Zariski open subset of a GIT quotient of ${\mu }^{-1}(0)$, where $\mu$ is a holomorphic moment map such that ${\mu }^{-1}(0)$ consists of the ADHM data. The purpose of the paper is to study the geometric properties of ${\mu }^{-1}(0)$ and its GIT quotient, such as complete intersection, irreducibility, reducedness and normality. If $K=USp(N\mathrm{/}2)$ then $\mu$ is flat and ${\mu }^{-1}(0)$ is an irreducible normal variety for any $n$ and even $N$. If $K=SO(N,R)$ the similar results are proven for low $n$ and $N$. As an application one can obtain a mathematical interpretation of the K-theoretic Nekrasov partition function of Nekrasov and Shadchin (2004).