Does that mean there's a limit to how much you can effectively upvote/downvote someone?
Limit, in the mathematical sense, yes. The more
tenured you are on the forums, the less and less individual votes affect you. You'll never reach the point of being immune to Trust ratings, but you can get damn close eventually.