Oyyy...
I know I shouldn't dip my toe in these waters. I really shouldn't.
But of course I'm going to anyway.
There are really two arguments here, although it doesn't seem like it. Yes, you're absolutely right - you're justified in dating who you want, and you should choose preference. I chose preference, too - I prefer White men. But if a Black man or an Asian man or any other Man has the characteristics I'm looking for, I'd date him. Would I date a woman? No. But I doubt a woman would be offended if I chose not to date her. There are plenty of females in the sea who would, after all.
On the other hand, let me flip this a little bit. Being Black has typically been seen as "less than" in the American culture. It hasn't been seen as an equal race say, to being White. All things being equal, if it *were* an equal exchange, then there wouldn't be any problem. But because African American people have had problems with acceptance, it's difficult to see a turn-down as equitable.
It's kind of like a man who is bald or short who keeps hearing women say, "Oh, I'd never date a bald man." or, "I'd never date a short man." Sure it's preference, but it still stings because you keep hearing it and you know it's uncomfortable to hear. You feel self conscious about it.
Anything that's considered "weak" in society as far as dating is concerned will be reflected as offensive or uncomfortable to the party who was rejected. Sure, inevitably, you'll find the person who doesn't care. But there are fewer people to choose from and you get tired of feeling rejected by those who haven't even considered the possibility.
It's not prejudiced or racist if you have considered the idea and have just chosen preference. It *is* prejudiced or racist, however, if you won't even give the possibility a chance and you've chosen never to consider it as a possibility.