Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

How are the crawlers identifying themselves? If it's user agent strings then they can be faked. If it's cryptographically secured then you create a situation where newcomers can't get into the market.


Google publishes the ip addresses that google bot uses. If someone claims to be google bot but is not from one of those addresses, it’s a fake.


And in that case both systems end up with a situation new entrants can't enter.


I don't see how that helps the case where the UA looks like a normal browser and the source IP looks residential.


How about if they claim to be google chrome running on windows 11, from a residential IP address? Is that a human or an AI bot?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: