Even the default example of "Hello, how are you?" from English to French yields an awfully wrong result ("Hello, what is your experience?")...
I wouldn't trust them for anything else.
The other models are not better, here's the text generation output from "I enjoy walking my cute dog":
> I enjoy walking with my cute dog, I have been going to the park, and I just happened to like walking with my cute dog. I like to play with the dog.
My dog (Hannah) has been on my way home since December and when she came home she told me to go out and stay back. I told her that she had been too busy. I had to start working and had to go outside and go see myself again.
It could be just an algorithm that generates random sentences that it wouldn't make less sense.
I think it's worth pointing out that the library just gets the models working in the browser. The correctness of the translation is dependent on the model itself.
If you run the model using HuggingFace's python library, you will also get the same results (I've tested it, since, I wasn't too happy with those default translations and generations).
With regards to the text generation output, this is also similar to what you will get from the PyTorch model. Check out this blog post from HuggingFace themselves which discusses this: https://huggingface.co/blog/how-to-generate.
> Even the default example of "Hello, how are you?" from English to French yields an awfully wrong result ("Hello, what is your experience?")...
Really? For me that gives "Bonjour, comment êtes-vous?" with the default settings.
> text generation output
Yeah, text generation is really something that requires a big model. The Llama 7B param model quantized to 4bit is 13G and that is the smallest model I'd actually attempt to use for unconstrained text generation.
« Bonjour, comment êtes-vous? » barely translates to « Hi, how are you feeling today? » or, depending on the context, to something like « Hi, please describe yourself » to a native French speaker.
I wouldn't trust them for anything else.
The other models are not better, here's the text generation output from "I enjoy walking my cute dog":
> I enjoy walking with my cute dog, I have been going to the park, and I just happened to like walking with my cute dog. I like to play with the dog. My dog (Hannah) has been on my way home since December and when she came home she told me to go out and stay back. I told her that she had been too busy. I had to start working and had to go outside and go see myself again.
It could be just an algorithm that generates random sentences that it wouldn't make less sense.