I don't want the legitimate uses prevented. The default behavior should be to prevent, and the legitimate uses should explicitly opt-in. That way, you only have to do security analysis for those explicit points.
This to me is a server side issue- but that doesn't necessarily mean it's on the app developer. The behavior you're talking about can be set most servers directly, by adding the "X-Frame-Options" header into every request by default. Then exceptions would have to be made explicitly, by either the server admin or application developer. If anyone should change the default behavior (which I am not convinced is the case) it should be the server developers, not the browsers.
CSRF is solved very simply by using tokens for each field. If the attacking site can't load the other page, it can't pull the token out, and without the token the post gets discarded. If you've abstracted your form generation this should be super simple to add.
Why shouldn't you trust user-provided data to secure the same user's data? The potential attack is someone forging their own referer header in order to attack themself.
What I don't get is how arbitrary cross-site POSTs with malicious values are allowed. As far as I can tell, anyone can post this form:
<form action="http://bank.com/send_money><input name="to_account" value="SCAMMER-1234"></form>
Worse, one article will tell you to only allow Referrer == "bank.com", and then another will tell you that even that is no longer enough?!!!
Why can't we change the browser or the web server layer to prevent this by default?!