If I had to guess, a tool like this might be useful in a shop with a lot of inexperienced devs. It's a thin wrapper to make sure everyone walks the same well worn path the same way. You have 2-3 devs work on the tooling, and a much larger team doing rote ETL.
I worked at a shop that did this and the trade-off is TTM, as your 2 person tools team is constantly needing to unblock ETL team with new features as they encounter new requirements in the wild.
If your ETL team is 20+ people and the tools team doesn't have a head start, tools team will quickly fall behind an insurmountable backlog as your ETL team spins its wheels. But you might save some money if you choose the right KPI..
I think this is the case: when you run your pipelines at scale you want to standardize and simplify some repeatable aspects to lower the cost of managing them. You may also want to be orthogonal to orchestrator engines (or triggering engines) and avoid getting too opinionated and inflexible in the future. So this framework is exploring some sweet spot between raw spark pipelines and low code etl engines.
yeah though a lot of these fall for a variant of the "universal standard" conceit joked about in xkcd. All these low-code solutions suck, so we'll build our own in-house that surely won't have the same pitfalls..
I built a data processing framework at GE that let junior devs write whatever code they needed to transform a particular input. It provided an interface that they had to satisfy (for data lineage metrics) but otherwise scaled their code without them having to understand the distributed architecture or anything about the platform. Exceptions flowed up to the platform and became part of the data lineage metrics.
I walked into 20 years of adhoc code that had zero data lineage, recoverability, or scalability that was breaking daily. There were contractors with over a decade of tenure whose job it was to troubleshoot and fix their own brittle processes (and make new ones) daily.
I got laid off (747Max plus pandemic) as I was rolling it out and they went back to the old way.
Subsequently, a new startup (Pantomath) emerged with former GE engineers (and other former colleagues of mine) from my former department to address that problem domain.
Based on my experience trying to socialize this type of solution, sales are going to be a bitch.
I worked at a shop that did this and the trade-off is TTM, as your 2 person tools team is constantly needing to unblock ETL team with new features as they encounter new requirements in the wild.
If your ETL team is 20+ people and the tools team doesn't have a head start, tools team will quickly fall behind an insurmountable backlog as your ETL team spins its wheels. But you might save some money if you choose the right KPI..