Don't prematurely optimize. Make your code work, make your code readable, and then profile your code.
Without profiling you're guessing where the most impactful work can be done. You don't want to spend a week shaving 150ms when you could have spent a day to make your program 10s faster.
Code quality is highly subjective though and often what some might consider as "high quality" also comes with a cost to performance.
Fortunately the linked book provides a simple way to decide when something needs to be optimized under "Understanding High Performance" [0]
--
Before we can create high-performance code, we must understand what high performance is. The objective (not always attained) in creating high-performance software is to make the software able to carry out its appointed tasks so rapidly that it responds instantaneously, as far as the user is concerned. In other words, high-performance code should ideally run so fast that any further improvement in the code would be pointless.
Notice that the above definition most emphatically does not say anything about making the software as fast as possible. It also does not say anything about using assembly language, or an optimizing compiler, or, for that matter, a compiler at all. It also doesn’t say anything about how the code was designed and written. What it does say is that high-performance code shouldn’t get in the user’s way—and that’s all.
Without profiling you're guessing where the most impactful work can be done. You don't want to spend a week shaving 150ms when you could have spent a day to make your program 10s faster.