The Westermarck effect attempts to explain why that is not a simple mental adjustment. It's a theory, but it states "people who live in close domestic proximity during the first few years of their lives become desensitized to sexual attraction." [1]

[1] https://en.wikipedia.org/wiki/Westermarck_effect