Honest question for you: are these puzzles actually a good way to test the models?

The answers are certainly in the training set, likely many times over.

I’d be curious to see performance on Bracket City, which was featured here on HN yesterday.