Distinct differences of singular moduli

Let $E_1,E_2\mathrm{/}\mathbb{C}$ be non-isomorphic elliptic curves with complex multiplication. We prove that the pair $(E_1,E_2)$ is characterised, up to isomorphism, by the difference $j(E_1)-j(E_2)$ of the respective $j$-invariants. In other words, we show that if $x_1,x_2,x_3,x_4$ are singular moduli such that $x_1-x_2=x_3-x_4$, then either $(x_1,x_2)=(x_3,x_4)$ or $(x_1,x_3)=(x_2,x_4)$.