Making it 'raw fast' is what compilers/JITs are for.
You still need to pick the right algorithms, but I've personally been really focusing on elegancy. Maintenance and Operation of code is the major costs of software, not writing new stuff -- so when you write new stuff, I believe that decreasing those other costs should be the first priority.
I agree completely. A common mistake is thinking that performance can be tacked-on later. Thinking about performance up-front can inform elegant designs.
Targeted performance improvements tend to make specific operations more efficient. Elegant architectures tend to completely eliminate huge swaths of duplicate and unnecessary operations, and so typically have the opportunity for much greater performance improvements.
You still need to pick the right algorithms, but I've personally been really focusing on elegancy. Maintenance and Operation of code is the major costs of software, not writing new stuff -- so when you write new stuff, I believe that decreasing those other costs should be the first priority.