Well I am not surprised that you are not treated well but I am surprised it's not racial. I am American and I apologize for what it is worth.. I think there is still quite a bit of lingering overreaction to 9/11.
No need to apologise, at all. It's not you that is making the situation and I honestly wouldn't bother apologising for those that are.
Problem is; it is only a minority (IMO anyway) and a fairly small one. But they tend to be vocal/obvious, and it is one of those things that doesn't take much to make you feel uncomfortable.
I don't think there is anywhere much that I like better than Manhattan! :)