I think it's kind of disingenuous maybe to claim such improvements in training efficiency when they rely on:

- Existing models for data pseudo-labelling

- ImageNet pretraining

- A frozen text encoder

- A frozen image encoder