It’s not just a problem for size, don’t forget you also have to copy bits across busses and several layers of cache - and doing so is not instant!

And that’s just thinking in terms of logical RAM. You then have translation tables which pointers need to index into - having them larger makes indexing even more slower, compounding the issue