I think you can downvote at some point, once you have enough karma

There is one, you need 500 karma to unlock it.

While you can't downvote me with so little karma, you can express your disagreement in a more thoughtful way, by pointing to a mistake in my argument. A downvote won't change my position, won't silence me and won't even make my comment disappear. So why bother?