I use GPT4 and it confabulates constantly. Try it on a language in the 10th-20th range of popularity (let alone lower) and I expect you'll see it as well. I don't see that as often with Python, the only top-5 language I currently work with.
I can confirm as well. I tried something with GNU APL ... and the answers were purely ridiculous. It fabulated some totally wrong stuff based on Dyalog APL with invented libraries and more.
Okay, I didn't expect usable results here but everyone should be aware... when even non-mainstream technologies are being discriminated, what about the human spectrum?
Right, that's my experience as well. Python, C, C++, bash, PowerShell, etc, are all pretty much spot on 95/% of the time. It will start to fumble if I specify PS 4.0, or Python 2, C89, etc. Or use X language but don't use popular library Y.
In my experience it depends a lot more on what you prompt it with rather than the language itself. As soon as you task it with something where your prompt doesn’t hit the right marks, I’ll go into its “inventive” mode and simply make stuff up. Copilot for enterprise is a little better in terms of not making things up, but the flip side of this is that it often simply provides a very, very, useless result.
My experience with it for Powershell hasn’t been as impressive as yours. It simply invented functions for the PnP module. Which I suppose isn’t in too much use these days, especially not against on-prem SharePoint. I’m not an expert on Powershell, however, and I really don’t think the documentation on things like PnP search queries is very good. So this was an area where I got to experience what it’s like to use these tools when you’re not the expert. In the end it was trial and error along with various shitty internet articles that helped me.
That being said. You can easily have GPT4 solve your CS exam questions, and likely whatever technical interview you’re given if you’re applying at jobs which do these things. And as such, I guess you could also argue that a lot of what we teach in CS is now even more dated than before. Because even if some of it is sort of useful for basic understanding, I’d bet money on students “cheating” their way through where they can. Because why wouldn’t they use the “calculator”?
