Beam tube boundary effects in stray light modeling of long Fabry-Perot arm cavities for third-generation gravitational-wave detectors

Next-generation gravitational-wave detectors such as Cosmic Explorer and the Einstein Telescope will operate 10-40 km Fabry-Perot arm cavities inside vacuum beam tubes. FFT-based paraxial tools treat propagation in free space and therefore do not explicitly enforce beam tube boundary conditions. We introduce a waveguide-like mode description of the optical field that incorporates an imposed beam tube boundary condition and enables an independent benchmark of free-space FFT tools We derive the associated modal-mixing matrices for mirrors and baffles, including a closed-form series for axisymmetric circular apertures. We quantify the strain-equivalent couplings from baffle miscentering and from a localized near-wall tube defect, and show that they are suppressed as baffle density increases. In the relevant regime of densely baffled cavities and small perturbations, beam tube boundary effects are subdominant, which supports the continued use of FFT-based codes to guide the design of 3G detectors.