Anecdotally Claude is just as bad as every other LLM.

Step into more niche areas e.g. I am trying to use it with Scala macros and at least 90% of the time it is giving code that either (a) fails to compile or (b) is just complete gibberish.

And at no point ever has it said it didn't know something.

Yep, get into any sufficiently deep niche (i.e. actually almost any non-trivial app) and the LLM magic fades off.

Yeah sure you can make a pong clone in html/js and that's mainly because there the internet is full of pong clone demos. Ask how to constraint a statsmodels lineal model in some non-standard way? It will gaslight how it is possible and make you loss time in the process.

Making a pong clone by telling the LLM to make a pong clone is a cute trick that sometimes works, but that's not the way anyone who understands how to properly use these tools is using them. You don't describe and app and hope the LLM builds it correctly. You have to know how to architect an application and you use the LLM to build small pieces of code. For example, you tell it to build a function that does x, takes the inputs a, b, and c and returns z.

LLMs don't turn non-coders into coders. It gives actual coders superpowers.

No true scottsman fallacy. I know how to use them, but using them "correctly" still produces many errors.

They suck at non-trivial code outside of standard library usage and boilerplate coding: I gave an example and parent did as well. In that regard would at least change your phrase from "actual coders" to "actual senior coders", as any junior receiving bad advice (in eternal loops as LLMs normally like to do it) is only going to make them waste time and tokens.

My point is that while you do have to give them coding problems that would have appeared in their training set (I guess you could call that trivial), every coding problem becomes trivial when you break it down to it's constituent parts. As you know, the biggest applications are just a lot of very simple building blocks working together. The point of using LLMs to code is not to solve complex problems. It's just to write code you could have written yourself at the speed of light using a natural language interface.

The way you described using LLMs to code seems like the approach someone who doesn't know how to build software might take, which is why I used the wording I did. From that angle, I agree with you - I can't even get Sonnet to create a working prototype of a basic game from a prompt. That said, I'm using it to build a far more complex enterprise web app step by step by using it in the way I mentioned above. It does work for these things, but you have to already know how to do what the LLM is doing.

I mentioned the pong example because that is what non-coders LLM users show and what the industry is proposing as the future of software development: no coding experience necessary.

> It does work for these things, but you have to already know how to do what the LLM is doing.

Yes, we totally agree. But even then, using models "correctly" in my experience and breaking down the problems for them gets you so far, once you start using weird/niche APIs (probably even your own APIs when your project gets big enough and you are not working with much boilerplate anymore) the LLM will start getting single concepts wrong.

And don't get me wrong, I understand those as limitations of a tech that still is immensely useful in the correct hands. My only issue with that is how these products are actually being marketed: as junior devs copilots or even replacements.

As a coder with some noncoder friends who have made some very impressive things with chatGPT, you're selling it short.

It does both. It gives coders superpowers, and gives noncoders the ability to do things that would have previously taken them months, or another person.

Do you mind sharing what they've created with it?

They created a touchscreen GUI in tkinter with more-than-trivial behavior to use as a frontend for input for a device they created. They were able to describe what they wanted, and in less than two hours have it working. This is someone with no software experience.

Three years ago, if I had been asked to create something like that, it would have taken me more than two hours, just because I've never used tkinter and would have to spend time reading the docs and figuring out how to make the different input boxes and laying them out properly.

I looked at the code, and no, it's not great. It's not designed "well" and isn't very extensible. But it works for him, doesn't need to be extended, and all in half a morning.

Not even close. I’m a programmer but also a guitarist. I love asking it to tab out songs for me or asking it how many bars are in the intro of a song. It convincingly gives an answer that is always way off the mark.