If I were you, I would look into writing equation and graphing calculator plugins for an existing animation tool, such as Blender.

I'm imagining something like a graphing calculator where you can define parameter variables that become interactive sliders, or that can be scripted or controlled by animation curves to produce baked animations.

I've been using Desmos for a year now, it's great! Waiting for the 3D version.

Although you can write your own 3D renderer inside Desmos...

Like the Manipulate command in Mathematica?: https://reference.wolfram.com/language/tutorial/Introduction...

I would start by looking at the current space of math software: TeX and LaTeX, MathJax, R, Wolfram Alpha, GNU Octave, MATLAB, Microsoft Excel, Omnigraffle. Plus decades of bespoke math educational video games (originally distributed on floppy/cd-rom; later online flash games).

Then think about what subdomain of mathematics you want to write software for. (9th grade polynomials, graphing, and algebra? Addition? Times tables?)

I'd also think about who you expect to use the software. (Just you? Teachers? Students?)

My dad used to work in a TV station's meteorology department. He is also a math teacher. He used some of the animation software (running on Irix on SGI O2's at the time, if I remember right) and green screen for weather reports to make math videos for his students, long before YouTube.

I was able to pick up that software and start making animations during a tour I got of the studio. The tools were designed to allow linking objects, paths, and animations in a very intuitive way.

See if you can find a demo of some of those weather animation user interfaces as there is probably good inspiration there.

Do you remember the name of the animation software they used?

“Looking at” R, Matlab and sage alone is easily year worth of work.

Probably like Mathematica crossed with After Effects.