What bothers me is where this leads. It leads to exclusive groups, insider clubs, and the self-perpetuating belief that the elites who run the world are better than others

On the contrary, they are not . Elitism stresses more on social status and this is far from being healthy for any society because it would then mean that those without social status sucks and are therefore not good enough.