I'm as skeptical as anyone, but have you ever heard of companies like Oracle, which got rich off a database or Snowflake (current market cap 65B)? Companies pay oodles of money for that capabilities.

oracle succeeded because of its lobbyists and sales contacts, so much so that they spun out into another multi billion $ org

I’d imagine pretty much all of the s and p 500 companies rely on databricks, a large percentage of them at least

because it's recommended by nearly all consultants and Microsoft.

Simple as that, it's consulting Heaven. Much like SAS and SAP. Everybody happy. Now to be far to databricks, if used properly and ignore the cost, it does actually function pretty well. Compared to Synapse, PowerBI Tabular, Fabric, Azure ML, ... that's already a big big big step forward.

Databricks is janky, but so much better than the Azure services for data.

If you're buying from microsoft, it won't be cheap either way, might as well treat yourself a little bit.

Dont forget salesforce

Spark