I came up with the idea of independent flagging pool, Here's something I posted about it.

https://steemit.com/steem/@transisto/separate-downvoting-power-pool-concept-visualized

There is only one user involved when down-voting so there is no "crowd" per se.