Geodesics in the Deep Linear Network

We derive a general system of ODEs and associated explicit solutions in a special case for geodesics between full rank matrices in the deep linear network geometry. In the process, we find horizontal straight lines in the invariant balanced manifold that remain geodesics under Riemannian submersion.