> What about an extensible format that would have as part of header an algorithm (in some recognized DSL) of how to decompress it (or any other step required for image manipulation)?

Will sooner or later be used to implement RCEs. Even if you could do a restriction as is done for eBPF, that code still has to execute.

Best would be not to extend it.