You’re right in the sense that people often use the term imprecisely and hyperbolically, but in this discussion they are more right than wrong, at least by this measure: https://en.wikipedia.org/wiki/Oligarchy#Putative_oligarchies

> That’s entirely different than determining agricultural policy, or putting people in jail, or conducting the census, or maintaining the border, or doing anything else that a ruler does

You’re making a mistake of your own by conflating oligarchy with tyranny. They often go hand in hand, with the former generally preceding the latter. So it’s probably better to cry oligarchy before it’s a given rather than afterwards.