You are right. Most of the voting bots have high reputations and a ton of SP so as @mindhunter has pointed out it is definitely skewing the reputation system.