Hessian Operators on Constraint Manifolds

On a constraint manifold we give an explicit formula for the Hessian matrix of a cost function that involves the Hessian matrix of a prolonged function and the Hessian matrices of the constraint functions. We give an explicit formula for the case of the orthogonal group $\mathbf{O}(n)$ by using only Euclidean coordinates on ${\mathbb{R}}^{n^2}$. An optimization problem on $\mathbf{S}\mathbf{O}(3)$ is completely carried out. Its applications to nonlinear stability problems are also analyzed.