Weird how we ended up making real-life objects that "curse" everything and everyone they touch with "invisible evil death magic". It's right out of a fairy tale.
I think as well as the obvious connection to nuclear warfare, part of the public fear of nuclear technology comes directly from how highly radioactive objects mimic the idea of a "curse".
In reality there's many forms of industrial pollution that are arguably scarier than nuclear waste, but none are quite so eerie in how their harm is caused. Even though it represented the very worst practices for these materials, the idea of a place like Lake Karachay where even half an hour on its shore would have killed you without you even knowing you were doomed is really unsettling. I don't think nuclear technology should be avoided, but there's definitely a formidable image problem here I think.
Chemical pollution is a lot scarier in a way, since it's much harder to detect. Even absolutely tiny amounts of radioactive contamination its existence to cheap widely available detectors. But I think chemical pollution is easier to ignore since it's harder to detect.
A lot of chemical pollution lasts literally forever, too. It's kind of funny the way the mind works. "This will remain dangerous for ten thousand years" somehow sounds scarier than "this is always dangerous."
Germs are already a bit like that. A lot of these superstitions were not bad for infection control, even if the underlying mechanism was not understood (much like the Standard Model, incidentally).
Unless you work with lead or eat off of leads plates all day, the risk of exposure is low. Even lead pipes are not as scary once they have been mineralized. Same with asbestos, it's the workers that handle it that are at the most risk. An asbestos tile or insulation just sitting there is not a huge risk.