.. it could be that the inactive accounts never bothered to vote for witnesses in the first place?

I would guess so, either through lack of knowledge or for a reason.

I too think it's a little early to look into removing 'inactive' account witness votes, perhaps in a few years time...