Graham isn't a conservative. He consistently splits from other Republicans. He's criticizing his own party as a party of "angry white guys" when he says this. Surely you don't think Graham considers himself on the same side as "angry white guys"! The article linked does not quote him as an example of racism, but rather as someone inside the Republican Party who is criticizing their own demographic.