Liking music does not make you special or smarter or better than other people. Liking music is simply how your individual specialness surfaces and expresses itself. Other people who like other things use those other things to express their specialness and their perspective on the world and their personal relationship to it.
I catch myself thinking this sometimes as well. It’s bull.
You are not better than other people because you are musical, any more than they are better than you because they can slam dunk, love math, create video games, dance a two-step, paint, write poetry, or collect action figures. These things are just different ways of doing the same thing, which we all do: express ourselves.
Certainly there are some people out there who are dense, nasty, narrow-minded, what have you. But that has nothing to do with how much they happen to like your hobbies.
You are not better than anyone else. You are not better than your audience. If you think this, even in a deeply hidden place, it will poison whatever message you try to send through your art to the point where you might as well not bother doing your art because you’re doing damage through it.