> Since a user requested the fetch, this fetcher generally ignores robots.txt rules.
Normally the expecation is that the user-agent faithfully presents the content it fetched.
If I make a browser that fetches bbc.com, and strips away ads and presented it to users - I would expect BBC to not like it and block the user-agent from accessing it. It isnt a robots.txt thing. It is a user-agent thing.
Normally the expecation is that the user-agent faithfully presents the content it fetched.
If I make a browser that fetches bbc.com, and strips away ads and presented it to users - I would expect BBC to not like it and block the user-agent from accessing it. It isnt a robots.txt thing. It is a user-agent thing.