That word does not mean what you think it means.

In fact, today it doesn't mean anything, because it means whatever the speaker wants it to mean. It's used simply to denigrate whoever the speaker dislikes or disagrees with, without making a rational counter-argument.

    Speaking a language != being a member of a race

    Preferring a language != being prejudiced against members of a race

    "That's racist!" != it being racist

So, a native english speaker proposing all countries drop their languages and adopt english wouldn't be racist to you -- but someone calling that out, would?

I'm saying it shouldn't matter if the person proposing English as a world language is a native English speaker or not.

Aside, this "vantage point" logic seems to be similar to the logic that only some people (white people, for example) can be racist.