Movement is implemented by using approximate numerical solutions to the geodesic equations (a system a differential equations whose solutions give geodesics in a Riemannian manifold). As such, there are some numerical inaccuracies due both to an imprecise solution to the equations and (I can only assume) due to accumulation errors from adding thousands of floating points together.
In general, ODE solvers can't be expected to maintain invariants of the ODE system, such as periodicity. This is typically handled by choosing a solver that maintains such invariants by construction, e.g., symplectic methods for Hamiltonian systems conserve symplectic 2-form. Adding many floating-point numbers together causes serious problems only in relatively rare cases, unlikely to occur here (e.g., see Higham's accuracy and stability book).
If I were to guess, the issue is maybe here (https://github.com/imeckler/diffgeo/blob/master/Main.elm#L45...), where the direction change is applied when the key is pressed, which means that the direction change doesn't happen inside the ODE itself: in effect this might be solving "go straight-tiny turn-go straight-tiny turn...", which would not be periodic even with perfectly exact solutions. If this is the case, this is like applying a first-order splitting method in which turns are handled with the forward Euler method, which would be quite inaccurate.
An extra source of error would be a positive Lyapunov exponent of the geodesic flow. When it is positive, the errors in the geodesic ODE solutions accumulate, causing exponentially large (in time) changes, meaning that the accumulated errors are much more noticeable than in geometries with non-positive Lyapunov exponents of geodesics (e.g., flat geometry would have smaller errors).
That said, it's not clear that you should actually expect to get periodic case in every case like this. I just kind of assumed they would be periodic based on how close they got. Still, flat circles seem to deviate after about half a dozen orbits. (Edit: With a bit of further experimenting, it seems curves in the hyperbolic upper half-plane that constantly turn should be periodic, so there's probably a bug.)
A friend of mind suggested the very good idea of solving the diffeq for the constantly turning curve and using this when the user is simultaneously turning and going forward instead of doing one or the other first.
Another thing you could do is add an analog input ('virtual joystick') in the corner. Basically, a little box which someone could touch/click inside of to start moving the arrow with a specific turning radius + velocity.
