That’s definitely a contentious thing to say. I am difficult to offend, but I like to give people some benefit of doubt to explain that/why something is problematic. Where that gets tricky is that many people respond by feeling personally hurt, even by comments which weren’t directed at them. That’s their right, I suppose, but I think it makes the process hard to learn from.
The worst of it I think is not being disliked or avoided for one’s actual words, but when people tell you that it is because of what those words - and extension, you - represent to them. Because I can take responsibility for what I say, but not how others may choose to interpret this. Sometimes I can guess, sometimes I cannot. There is no defense against “you might as well have said”. With many, maintaining their preferred ingroup/outgroup boundary is far more important than open discussion, because the former feels safely insular, and there are no guarantees that the latter won’t go in complicated or awkward directions. So the compromise many make is to have a sort of cultural/subcultural “script” of token positions, and anything outside of that is more likely to arouse suspicions about one’s character, rather than curiousity about the nuances of one’s views. There probably is no easy answer!