I use a gaiter for this and it's crucial. It gets damp with breathing, haven't really figured out a good solution besides spinning the gaiter around every once in a while.
I switched from a gaiter to a scarf precisely because of the dampness.
I can tie a scarf to be just slightly looser than the gaiter, and the extra space decreases the dampness massively, while still retaining almost all of the warmth.
