Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

You could do this without effecting normal traffic depending on uniqueness of ip doing the scraping.

Love the idea.



I think you missed the point - if people show up at $PROXY expect nice stuff but see junk, then they won't move over to $REAL and instead blame $REAL.

E.g. you'd like some way to redirect people from $PROXY site to $REAL site, and disgusting content on $PROXY won't do that - it'll reflect poorly on $REAL


If you can identify the crawler - you can provide 'dynamic' content for that specific user context.


It's a proxy, so there's no "crawler". It's just an agent relaying to the user. Passing something to this proxy agent just passes it directly to the user.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: