You're also assuming that their town doesn't have a lot of African immigrants living on its south side, and that the speaker wasn't (very questionably) saying that their experience with Africans affected the way they interacted with local black people.