It is annoying that people abuse downvotes.
I think downvoting based entirely on opinions is wrong and should definitely not be free.
It will always be necessary for abusers and trolls.
In the future I hope to see it more more regulated, but for now there are serious issues like vote trading and vote selling going on.

Maybe later allow it to be cancelled by using downvote power to negate downvotes.

It is just is unfortunate for people who are legitimate but may be writing stuff others don't like or want censored.