General monotone formula for homogeneous $k$-Hessian equation in the exterior domain and its applications

In this paper, we deal with an overdetermined problem for the $k$-Hessian equation ($1\le k<\frac{n}{2}$) in the exterior domain and prove the corresponding ball characterizations. Since that Weinberger type approach seems to fail to solve the problem, we give a new perspective to solve exterior overdetermined problem by combining two integral identities and geometric inequalities inspired by Brandolini-Nitsch-Salani’s results \cite{BNS}. Meanwhile, we establish general monotone formulas to derive geometric inequalities related to $k$-admissible solution $u$ in ${\mathbb{R}}^n\setminus \mathrm{\Omega }$, where $\mathrm{\Omega }$ is smooth, $k$-convex and star-shaped domain, which constructed by Ma-Zhang\cite{MZ} and Xiao\cite{xiao}.