Modular curves $X_0(N)$ of density degree $5$

We determine all modular curves $X_0(N)$ with density degree $5$, i.e. all curves $X_0(N)$ with infinitely many points of degree $5$ and only finitely many points of degree $d\le 4$. As a consequence, the problem of determining all curves $X_0(N)$ with infinitely many points of degree $5$ remains open for only $30$ levels $N$.