Anti-self-dual instantons with Lagrangian boundary conditions I : Elliptic theory

We study a nonlocal boundary value problem for anti-self-dual instantons on 4-manifolds with a space-time splitting of the boundary. The model case is $\R \times Y$,where$Y$isacompactoriented3-manifoldwithboundary$\Sigma$. The restriction of the instanton to each time slice $t\times \mathrm{\Sigma }$ is required to lie in a fixed (singular) Lagrangian submanifold of the moduli space of flat connections over $\mathrm{\Sigma }$. We establish the basic regularity and compactness properties (assuming $L^p$-bounds on the curvature) as well as the Fredholm theory in a compact model case. The motivation for studying this boundary value problem lies in the construction of instanton Floer homology for 3-manifolds with boundary. The present paper is part of a program proposed by Salamon for the proof of the Atiyah-Floer conjecture for homology-3-spheres.