I don't claim expertise in math or CS, but the book that sparked my intellectual curiosity was GEB. I read part way through it in high school and was really grabbed by the concept of recursion (brand new to me at the time). I ended up carrying that spark with me until it consumed me enough to change careers, from teaching music to programming.