I wonder how hard (or necessary) it would be to check "Follower Activeness"

I've thought about that, too (great minds think alike? 😉). Also median SP of followers. I think both values might be useful, but the only ways I know to implement them would be to query each follower account separately at the time of voting, or to store it in my own database and update it regularly. Both implementations require more compute than I want to put into it right now.