If an algorithm runs faster on CPU than GPU, then it's only a matter of time before it gets optimized for GPU and at the end of the day will still run faster on GPU.
At the very least, generic types of parallel training are applicable to these models (e.g. an ensemble of models that see batches in a different order or are initialized differently).
