> This bit isn't being optimized, it's being dragged out.

Of course, it is being optimized. People are working on increasing the sample efficiency. A simple search on Google Scholar will confirm it.