I may be mistaken, but I actually remember Kent Beck saying it in his original TDD book. That was the thing that made me understand how refactoring can take the place of big design up front. The best part is that you only work on things that you need as opposed to wandering around changing every piece of code that you don't like ;-)
This is my favorite bit. Katrina Owen mentions this in her talk on refactoring. "Make the change easy; then make the easy change."
https://www.youtube.com/watch?v=59YClXmkCVM