I guess the thing that bothers me is that there's this whole stereotype out there that rap is bad, vulgar, etc., and I think it's based on a very narrow view of the genre. Of course, I certainly can't blame anyone for having that opinion of rap based on what is promoted by the media, and you're right, it shouldn't affect my appreciation of the music.
BTW people might not know this since I haven't posted here much, but rap is just one of many genres I listen to, I listen to a lot more rock as well as folk, jazz, electronic, blues, etc.