My experience with coding-oriented LLMs has been that they fail pretty spectacularly on non-popular languages and are incapable of fixing errors in their own code.
I've wrestled for many hours trying to get them to write me a short, simple function in a non-popular language and they failed again and again.
I also suspect these LLMs are going to have huge problems when APIs and language definitions change, because the data they're trained on usually doesn't have API/language version metadata attached to it, so they have no way of knowing which version is applicable to whatever you're trying to do.
They may spit out code in an obsolete API version, for example, or even give you code containing mixed API versions. This will be a nightmare to debug.
Where they do shine for me is for repetitive tasks.
Say I am starting to write a list and start it out with "item 1", "item 2".. it will then let me tab complete some more items like value completion in e.g. excel.
It has definitely saved me time.
I've wrestled for many hours trying to get them to write me a short, simple function in a non-popular language and they failed again and again.
I also suspect these LLMs are going to have huge problems when APIs and language definitions change, because the data they're trained on usually doesn't have API/language version metadata attached to it, so they have no way of knowing which version is applicable to whatever you're trying to do.
They may spit out code in an obsolete API version, for example, or even give you code containing mixed API versions. This will be a nightmare to debug.