I do it maybe once a month to fetch <1000 URLs. I do it from my home PC with my internet connection. I was just using puppeteer (headless chromium), I will try making it use my own normal browser instance instead of the built-in one.

Thanks for the tips!