Is this just not a problem in practice, or is there some technical barrier I'm not thinking of here? I'm imagining a system where you say "hey Google, I'm about to post content X" that you do right before it goes live on your site. No race condition.
If you ping Bing or Google before you publish, they’ll get a 404 and will take that as a sign that there is no content there. They also will wait longer before trying to reindex a page that previously returned a 404.
Make a visible but unlisted URL, ping search engines, wait 5 minutes, list / link the URL from the home page of your site, publish the URL to RSS. Solves the 404 problem, doesn't it?
