Fortunately, real world does not run on boolean logic. "X, who is an authority on the topic says Y" does not prove Y is true, but provides evidence in favour of Y being true.
Sure, I agree to some extent.
In the hypothetical case it may be reasonable to assume that X knows what they are talking about; one can reasonably assume that they are better informed regarding the subject than a layperson.
But the reason for it being a fallacy is that this sort of argument doesn't provide evidence of any kind. Rather, it makes the assumption that it must be true, because the authority says so.