Only white asbestos (also known as Chrysotile) --- and the great number of people who have been exposed to large amounts of it and lived long healthy lives certainly makes one wonder; AFAIK it's only the other types, particularly blue asbestos, which are unanimously carcinogenic and the industry stopped using them a long time ago, but the effects from that (and contamination of white asbestos with the other types) have continued and led the majority of fears.
In the case of Mesothelioma I believe there's a fair amount of evidence that you need both : 1) Asbestos exposure and 2) to be a smoker. This might explain why some people exposed to asbestos remain healthy.
(On the latter two, the studies mostly have the correct outcome. What is difficult is convincing half of the American population that that matters.)