I like to leave spare machines running chess engines to analyze openings etc. You could also leave it churning for days on end on various Kaggle competitions (especially if it had a relevant GPU). Could also get the thing scraping interesting datasets at a slow and respectful pace.
For basic learning on somewhat larger datasets (sub-TB), I have found that my 750 TI SC GPU has been very adequate; the only time I maxed it out was while doing "live training" of a tensorflow CNN from images generated from a driving simulator - so I was using part of the GPU for 3D rendering, and part of it for the training. This was for some "self-driving vehicle" experimentation...
But running the model against the same simulation (that is, having the model drive instead of watching me drive and learning) actually used much less GPU; while I wasn't getting stellar frame rates in either case, they were adequate for the purpose.
I'm just mentioning it, because many people assume you have to have a crazy amount of compute power with a GPU for ML tasks right off the bat (and the associated higher costs of such a system), when in reality that may not be the case (depending on what you are doing and how you are doing it), and you can get away a lesser system.
Where I ran into problems with my GPU, though, was in the case of training using larger datasets (that is, non-streaming) - I couldn't fit them all in the GPU's memory (I think the card only has 1 GB), and so I had to do a batch system - and while it worked well, it was a lot slower because of shuffling data to and from the GPU.
But for my learning purposes, it still did a fine job IMHO; of course, if I were doing such tasks for actual analysis or other "professional" (and paying) purposes, a better card would definitely be in order.
Often I'm just looking for some vaguely novel line in an opening that might be interesting to pursue in a game (I am not a strong player but love studying). I sometimes like seeing the difference in analysis between Stockish and Leela of real games (Magnus Carlsen attributes much of his recent form to trying to play more like AlphaZero with all sorts of positional pawn sacrifices etc). I've also tried to do some programmatic stuff around the analyses to automate things (i.e. find a move in a common line that nobody in my database has ever played but the computer says is best, or find lines where it looks like the opponent only has a single path through to stay in the game etc).
