Human drivers aren't really aware of the capacity or credentials of those directing traffic. Source: Sometimes I direct traffic. I'm part of a volunteer organization, taking orders from another volunteer organization, often eventually, yes, getting marching orders from police. People who are wearing vests may or may not be cops, construction workers, or volunteers like myself who have little to no oversight preventing them from putting the vest on and walking out into a busy street when they shouldn't.

At an event I was at recently, traffic control involved at least four different organizations, which had their own gear and their own attire, so you really can't count on everyone you see on the road looking the same either.

Additionally in terms of lack of credentials, there's a certification process for directing traffic (I've renewed mine twice), but there's a pretty non-zero chance that the person who has directed you around a construction site actually bothered to get that, and you'll probably never know that either.

Your other mistake is believing human drivers actually do what people directing them tell them to. Source: Sometimes I try to direct traffic.