Cartan images and $\mathrm{\ell }$-torsion points of elliptic curves with rational $j$-invariant

Let $\mathrm{\ell }$ be an odd prime and $d$ a positive integer. We determine when there exists a degree-$d$ number field $K$ and an elliptic curve $E\mathrm{/}K$ with $j(E)\in \mathbb{Q}\setminus \{0,1728\}$ for which $E(K{)}_{\mathrm{t}\mathrm{o}\mathrm{r}\mathrm{s}}$ contains a point of order $\mathrm{\ell }$. We also determine when there exists such a pair $(K,E)$ for which the image of the associated mod-$\mathrm{\ell }$ Galois representation is contained in a Cartan subgroup or its normalizer, conditionally on a conjecture of Sutherland. We do the same under the stronger assumption that $E$ is defined over $\mathbb{Q}$.