The advisor is the most crucial factor in deciding a PhD/postdoc program (you're not an undergrad anymore!). When people talk about the prestige factor at this level, they are talking about the top 20 vs. the university of nowhere, and even in those cases, it comes down to the advisor and their reputation in the field (for your case, all the other factors are the same). At the end of the day, as a PhD student, you rarely need the "school" resources as opposed to the care/advice/networking you get from your supervisor and the facilities in their labs (that's also true at the department level; why would you care if the department has access to a remote telescope if your research is in quantum optics?) So, pay no attention to the side details (weather, location, rankings, etc.) and focus solely on the best advisor you can find wherever they may be.
Also, keep in mind that all the top schools produce a much larger volume of grad/postdocs than will there ever be faculty positions in ANY STEM field (so most grads end up in non-related positions, and many qualified applicants are turned down). That implies that you definitely need people to recognize your name when applying for jobs, and one way to achieve that is to put your name on an article with a reputable person in the field or have their recommendation (or be smart enough to make a name for yourself). Even in that case, deciding based on reputation is not merely sufficient. You want to invest 6-7 best yrs of your life, so do you like how the advisor treats their grads/postdocs and runs the lab? Do you want them to be hands-on, or would you rather be independent (very important to consider this)? Do you even like your research project/topic?
I always say that choosing a PhD advisor is like "virtually" marrying someone. You have to check all the qualifications to see if you're a "perfect match" before committing to anything (also, you would use your advisor's networking post-graduate, so you will always have them in your academic life to some extent). If you do not believe me, you can check the profiles of the newly hired physics assistant professors or associate professors across all the US schools, and you will see that there's no pattern in the academic background (some have PhDs from European/Asian universities that you probably never heard of and some even have PhDs in another discipline like chem or bio); nevertheless, their advisors were famous in the field + their research directions were promising enough to get them hired/promoted + the faculty of that school think they can collaborate with them at any level. Also, when it comes to hiring a new faculty (which I believe is the most illustrious job in physics), there is a ton of politics you need to do (connections to the faculty of the said school, presenting the best case for future research directions based on your past research, getting your teaching profile ready, etc.), and at that point, no one cares where you got your PhD from. So, visit any lab you have in mind, talk to people, and check the overall environment of the lab before committing (in summary, would you be happy for the next 6-7 years of your life doing what you always loved to do?)
Remember that if you do not choose the "right" advisor, you will regret it to the extent that you may even grow to hate your research (it happened to me and indeed to a lot of my friends), and it takes a long time to get back on track even if you're in the best institution (some of my friends at top institutions end up doing TA for many years because of this).
For disclosure, I'm a physics PhD student at one of the so-called top physics institutions. I happen to be involved in a lot of politics behind the scenes (e.g., faculty hiring) and try so hard for faculty positions!
The advisor is the most crucial factor in deciding a PhD/postdoc program (you're not an undergrad anymore!). When people talk about the prestige factor at this level, they are talking about the top 20 vs. the university of nowhere, and even in those cases, it comes down to the advisor and their reputation in the field (for your case, all the other factors are the same). At the end of the day, as a PhD student, you rarely need the "school" resources as opposed to the care/advice/networking you get from your supervisor and the facilities in their labs (that's also true at the department level; why would you care if the department has access to a remote telescope if your research is in quantum optics?) So, pay no attention to the side details (weather, location, rankings, etc.) and focus solely on the best advisor you can find wherever they may be.
Also, keep in mind that all the top schools produce a much larger volume of grad/postdocs than will there ever be faculty positions in ANY STEM field (so most grads end up in non-related positions, and many qualified applicants are turned down). That implies that you definitely need people to recognize your name when applying for jobs, and one way to achieve that is to put your name on an article with a reputable person in the field or have their recommendation (or be smart enough to make a name for yourself). Even in that case, deciding based on reputation is not merely sufficient. You want to invest 6-7 best yrs of your life, so do you like how the advisor treats their grads/postdocs and runs the lab? Do you want them to be hands-on, or would you rather be independent (very important to consider this)? Do you even like your research project/topic?
I always say that choosing a PhD advisor is like "virtually" marrying someone. You have to check all the qualifications to see if you're a "perfect match" before committing to anything (also, you would use your advisor's networking post-graduate, so you will always have them in your academic life to some extent). If you do not believe me, you can check the profiles of the newly hired physics assistant professors or associate professors across all the US schools, and you will see that there's no pattern in the academic background (some have PhDs from European/Asian universities that you probably never heard of and some even have PhDs in another discipline like chem or bio); nevertheless, their advisors were famous in the field + their research directions were promising enough to get them hired/promoted + the faculty of that school think they can collaborate with them at any level. Also, when it comes to hiring a new faculty (which I believe is the most illustrious job in physics), there is a ton of politics you need to do (connections to the faculty of the said school, presenting the best case for future research directions based on your past research, getting your teaching profile ready, etc.), and at that point, no one cares where you got your PhD from. So, visit any lab you have in mind, talk to people, and check the overall environment of the lab before committing (in summary, would you be happy for the next 6-7 years of your life doing what you always loved to do?)
Remember that if you do not choose the "right" advisor, you will regret it to the extent that you may even grow to hate your research (it happened to me and indeed to a lot of my friends), and it takes a long time to get back on track even if you're in the best institution (some of my friends at top institutions end up doing TA for many years because of this).
For disclosure, I'm a physics PhD student at one of the so-called top physics institutions. I happen to be involved in a lot of politics behind the scenes (e.g., faculty hiring) and try so hard for faculty positions!