Downtown is where all the ferry landings were, and where a lot of the bridges were built after.

Midtown is where the rail terminals stop, so suburbanites can quickly walk to their jobs instead of paying a second fare for subway/bus/taxi.

There is no correlation between bedrock depth and height in Manhattan, as explained in the article of the grandparent comment. In fact, most of Downtown is landfill.