Were the 178k URLs that they scraped on 178k unique domains or a very few domains?
The tool they wrote looks like it would have been a DoS attack if the latter were the case. There appears to be no thought or consideration for the sites they scraped that had linked to them, no attempt to read robots.txt and see whether they should fetch the pages.
What they did was wrong in the first place, but they could try and do things better in their attempt to repair things. Instead they continue to act in a way that externalises any cost to them and continues to treat their service as somehow privileged enough to deserve such breaks.
How many of the 53k failures to pull and scrape pages were to do with the fact that the servers in question were under a DoS attack by RapGenius?
The tool they wrote looks like it would have been a DoS attack if the latter were the case. There appears to be no thought or consideration for the sites they scraped that had linked to them, no attempt to read robots.txt and see whether they should fetch the pages.
What they did was wrong in the first place, but they could try and do things better in their attempt to repair things. Instead they continue to act in a way that externalises any cost to them and continues to treat their service as somehow privileged enough to deserve such breaks.
How many of the 53k failures to pull and scrape pages were to do with the fact that the servers in question were under a DoS attack by RapGenius?