Generalized quadratic matrix manifolds: Riemannian geometry and computations

We study the set $\mathrm{G}\mathrm{q}(P,Q)$ of real $n\times p$ matrices $X$ satisfying $X^{\mathrm{\top }}QX=P$, where the nonsingular matrices $P$ and $Q$ are both symmetric or both skew-symmetric and the set is nonempty. Special choices of $P$ and $Q$ yield the Stiefel, generalized Stiefel, indefinite Stiefel, and symplectic Stiefel manifolds, as well as the quadratic matrix Lie groups. We prove that $\mathrm{G}\mathrm{q}(P,Q)$ is a closed embedded submanifold of ${\mathbb{R}}^{n\times p}$ and that it is compact exactly when $P$ and $Q$ are both symmetric and definite of the same sign. We derive closed-form expressions for its tangent space and, under a canonical-like Riemannian metric, for the normal space, the orthogonal projection, and the Riemannian gradient and Hessian. We extend the Cayley, quasi-geodesic, and polar retractions to $\mathrm{G}\mathrm{q}(P,Q)$, with explicit domains, vector transports, and closed-form second-order corrections. These objects are the ingredients of Riemannian first- and second-order methods. In trace minimization tests with symmetric positive-definite, skew-symmetric, and symmetric indefinite pairs $(P,Q)$, the Riemannian conjugate gradient and trust-region methods reach the same objective value with different retractions and keep the iterates on $\mathrm{G}\mathrm{q}(P,Q)$ up to rounding error.