There are a thousand other communities competing for users. This is an easy deal-breaker. Removing downvotes takes away the ability to hide spam and shitposts. If the users are downvoting “legitimate” discussion, it’s probably because the users don’t think it’s legitimate, and the system is working as intended.
Instead the admin suggests you should “report” them. If this site were ever to gain any traction, there’s no possible way the admin could sort through allll of those reports. That’s the entire point of the downvote: community self-moderation.