Optimal configuration and symmetry breaking phenomena in the composite membrane problem with fractional Laplacian

We consider the following eigenvalue optimization in the composite membrane problem with fractional Laplacian: given a bounded domain $\Omega\subset $\mathbb{R}$^n$,$\alpha>0$and$0<A<|\Omega|$,findasubset$D\subset \Omega$ofarea$A$suchthatthefirstDirichleteigenvalueoftheoperator$(-\Delta)^s+\alpha \chi_D$isassmallaspossible\mathrm{.}Thesolution$D$ is called as an optimal configuration for the data $(\mathrm{\Omega },\alpha ,A)$. Looking at the well-known extension definition for the fractional Laplacian, in the case $s=1\mathrm{/}2$ this is essentially the composite membrane problem for which the mass is concentrated at the boundary as one is trying to maximize the Steklov eigenvalue. We prove existence of solutions and study properties of optimal configuration $D$. This is a free boundary problem which could be formulated as a two-sided unstable obstacle problem. Moreover, we show that for some rotationally symmetric domains (thin annuli), the optimal configuration is not rotational symmetric, which implies the non-uniqueness of the optimal configuration $D$. On the other hand, we prove that for a convex domain $\mathrm{\Omega }$ having reflection symmetries, the optimal configuration possesses the same symmetries, which implies uniqueness of the optimal configuration $D$ in the ball case.