It's not nearly so academic (in fact, it's for middle-grade readers) but Brandon Sanderson's The Rithmatist bases its magic system on a unique take on some foundational geometric principles.