They seem to have had this problem in the past and decided to skirt around it by...

icebraining · on April 25, 2018

As per that post, they only ignored robots.txt for .gov and .mil sites.

aero-002 · on April 25, 2018

IA disallow in robots.txt will still block archive.org, the blog post was about ignoring parts that were meant for search engines.

klez · on April 25, 2018

Yes, but it also says

> We are now looking to do this more broadly.

That's the part I'm asking about.

icebraining · on April 25, 2018

Right, but it doesn't mean they reverted it, they are probably still looking into it.