Hill-type formula for Hamiltonian system with Lagrangian boundary conditions

In this paper, we build up Hill-type formula for linear Hamiltonian systems with Lagrangian boundary conditions, which include standard Neumann, Dirichlet boundary conditions. Such a kind of boundary conditions comes from the brake symmetry periodic orbits in $n$-body problem naturally. The Hill-type formula connects the infinite determinant of the Hessian of the action functional with the determinant of matrices which depend on the monodromy matrix and boundary conditions. Consequently, we derive the Krein-type trace formula and give nontrivial estimation for the eigenvalue problem. Combined with the Maslov-type index theory, we give some new stability criteria for the brake symmetry periodic solutions of Hamiltonian systems. As an application, we study the linear stability of elliptic relative equilibria in planar $3$-body problem.