The exact growing conditions for olive production aren’t common in the US, so most of the production comes from California - west of Sacramento and south along the San Joaquin river. There are a lot of barriers in bringing specialty crops to market related to know-how and contracting sale of product, so even in other areas where growth may be possible it may be infeasible.
Most likely the answer in many such examples is it needs cheap human labor. US seldom lacks anything in terms of natural resources and always comes down to this.
https://www.agmrc.org/commodities-products/fruits/olives
https://croplandcros.scinet.usda.gov/