I think it's the lack of heat (AKA infrared light) also reaching your body, which, when receiving bright visible light, is evolutionarily wired to expect heat coming from that same light source (e.g. the sun). Feels off for you to be cold and for it to be super bright.

I'm currently experimenting with 'emulating outdoors' in my office environment for health and productivity, and so far I've started with an 8,000 lux fanless corn LED (bought off Amazon) hanging from the wall and it's brilliant. I immediately feel 'happier' when I turn it on in the morning, every single time, but after a while, I agree, something feels off. I feel 'cold', temperature-wise.

I have read (no source for now, just from memory) that having bright white light indoors (e.g. fluorescent) without the presence of other light spectrum (including possibly natural UV) can actually increase cortisol.

So I think that more than just visible light is needed to complete the picture. Careful, low enough levels of UV and infrared that doesn't point directly at you but more 'above' you like the sun itself, may be the optimum to get some benefits but not cause dangerous skin or eye problems.