Likelihood Ratio Gradient Estimation for Steady-State Parameters

We consider a discrete-time Markov chain $\mathbf{\Phi }$ on a general state-space $\mathsf{X}$, whose transition probabilities are parameterized by a real-valued vector $\mathit{\theta }$. Under the assumption that $\mathbf{\Phi }$ is geometrically ergodic with corresponding stationary distribution $\pi (\mathit{\theta })$, we are interested in estimating the gradient $\mathrm{\nabla }\alpha (\mathit{\theta })$ of the steady-state expectation $$\alpha (\mathit{\theta })=\pi (\mathit{\theta })f\mathrm{.}$$ To this end, we first give sufficient conditions for the differentiability of $\alpha (\mathit{\theta })$ and for the calculation of its gradient via a sequence of finite horizon expectations. We then propose two different likelihood ratio estimators and analyze their limiting behavior.