Degrees of points with rational $j$-invariant on $X_0(n)$ and $X_1(n)$

We give a classification of the degrees of the points with rational $j$-invariant on the modular curves $X_0(n)$ and $X_1(n)$. The degrees which occur infinitely often are computed unconditionally, while those which occur finitely often are determined assuming a conjecture of Zywina. To achieve this, we define the notion of $\mathcal{H}$-closures of subgroups of ${\mathrm{GL}}_2(\widehat{\mathbb{Z}})$, and compute the ${\mathcal{B}}_0(n)$- and ${\mathcal{B}}_1(n)$-closures of images of Galois representations of elliptic curves defined over $\mathbb{Q}$. An application to computing the set of isolated $j$-invariants in $\mathbb{Q}$ is also given.