> Why do we even have linear physical and virtual addresses in the first place

Mmmh, there was a time when PC's had non-linear memory [1]

Coding for this was IIRC an effing nightmare.

[1] https://en.wikipedia.org/wiki/X86_memory_segmentation