Even parasites that are definitely detrimental are generally not especially bad for hosts that have coevolved with them. Hookworm in the American South was completely debilitating for the impoverished whites it infected, but basically a nuisance for the blacks. This is generally how parasites work.
So there would be much more cause for alarm if you believed that your face mites were a variety you weren't already adapted to.
One of the things I find most interesting about the whole N. Americanus affair is that hookworm was a public health disaster of colossal proportions, and the medical research that recognized it was a big, important advance... but medical treatment for the problem is mostly irrelevant. The (very successful) "medical" intervention that solved this problem was just convincing people to wear shoes.