I wouldn't worry so much about followers as number of upvotes from said followers. Of all of the followers I have, only a handful are active on this site, and even fewer upvote my content.