They can set metrics on quality, but those can be easily gamed. (E.g. measuring average TTFB for a site instead of the real wall time to show visible content for the user.)
Management has the power to say that something isn't good enough and make it a priority. They also have the power to hire employees or consultants if the current team isn't capable of doing it.
Project management _definitely_ has the power to dedicate time to fixing performance issues.
Metrics can be gamed, but certain metrics - such as time to interactive, and time to fully loaded - are fairly well in line with what users actually care about. Even if they're gamed, a project manager can say, "This still feels slow to use. Dedicate the next (sprint|cycle|month|whatever) to performance work."
They can set metrics on quality, but those can be easily gamed. (E.g. measuring average TTFB for a site instead of the real wall time to show visible content for the user.)