Yes, when I introduce my interests to new people these days I say "night-sky watching", instead of astronomy, just to avoid this problem.

"Astronomy? You don't really believe in all that nonsense do you? Jane, this guy believes those horoscopes in the paper!"

Sigh