I struggle with that a lot, too. Everytime I have to describe someone I go by the most obvious attribute a person has, which can be their skin colour at times. Although it shouldn't make any difference than going with for example hair/eye colour, it still does.
I even feel awkward when I call the group of friends of my flatmate Latinos as they are a bunch of people from different parts of Latin America. Even my flatmate calls them that. Just as I call my friends from uni friends from uni.