Is there much written about the architectural models that enable this kind of customisation. This is highly relevant to my life right now, and I'm struggling to find much to read about how to build a core product that's extensible by third parties. The Apache web server, with its request lifecycle and chained handlers, is my mental model for that. But there must be other models too, and I'd like to see them described in the hopes that one leaps out as a good fit for our product and market.

You are right; probably I have to state the question in a more precise way, which would be: why is it such that this model seems only to be used in huge "enterprisey" contexts? The model of course depends on a certain modularity of the software in question, but hasn't that been a core concept not just for the biggest players, but also the smaller shops for a while now?

Implementation costs a lot, not just in coding, but in all the extra work that goes into sales, gathering requirements, training etc. Whoever nominally pays for all that work, it translates into a high price tag for the customer. Only large enterprises will get enough value from the customisations for that high price tag to be worthwhile.

All else being equal, a smaller company will spend a lower absolute amount to work around imperfections, and get lower value from any product overall. So of course they will not be willing to pay as high a price. Hence, for commodity use cases, use commodity software rather than fully/semi-customised.