On the sensitivity to hyperparameters of a Bayesian method to infer selection signatures

Motivated by a real-life problem, namely, inferring genomic regions affected by selection, we studied some properties of a two-step method that performs this task based on the FST. This approach exhibits computational simplicity as an appealing feature. Step one involves fitting a conjugate Bayesian model to find point estimates of FST and selecting the markers to be considered in step two, where they are grouped using such estimates. Subjective priors are frequently used in this problem and large hyperparameters may be encountered; hence, there is interest in addressing their impact. Preliminary analyses suggested that, the number of markers selected in step one tends to zero when the hyperparameters have the same value and tend to infinity, or are different but tend to infinity and their difference tends to zero. Thus, we studied how hyperparameters affect the posterior mean, variance, and predictive density of allele frequencies, and the posterior distribution of FST, formally. Moreover, an empirical approach based on simulated and real data was performed. Our main findings are summarized in two theorems that set the asymptotic behavior of the method, providing a foundation to understand the aforementioned phenomena. As to the empirical studies, the groups created in the second step, the estimated posterior means and the number of selected markers showed high sensitivity to hyperparameters. Albeit the mathematical intractability of the posterior distribution of FST, our results revealed asymptotic properties explaining two features of the method that lacked a formal elucidation.