A couple of different ways:

1. reputation - your tag votes would hold more weight the higher your reputation. Also, all votes after your vote on a tad would increase your reputation
2. -1 votes - of you think a tag does not relate to a post you can -1 it. If your reputation is greater than those before you then the tag goes into - territory

So is not th necessarily the number of votes but the number of reputation-weighted votes.

There could be restrictions on voting on tags and adding tags below a certain reputation.