Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

WGET can be pretty trivially told to send custom headers.


Try to do that to a site with CF bot protection cranked up... Not happening without a custom build/custom ssl proxy that mimics the SSL fingerprint of Chrome.


CF blocks you hard simply for enabling "do not track" in settings. The discussion about how awful they are needs to be had.


I haven’t seen a custom build of Wget, but for Curl there is curl-impersonate[1].

[1] https://github.com/lwthiker/curl-impersonate


It would be a lot of work to make it mimic a common profile though.


That work was probably done once, years ago. Might need a few string tweaks every few years, which could be automated.


No, because any “RMS” set of headers would only be shared by the small number of nerds who care, fingerprinting us more accurately again.


Just setup a honey pot and use headers from there ;)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: