I believe so. In my experience, you need to have that gut intuition (or experience) to say, “No way. That’s totally wrong.”

Since AI will capitulate and give you whatever you want.

You also have to learn how to ask without suggesting because it will take whatever you give it and agree.

Yep. I think a default state of skepticism is an absolute necessity when working with these tools.

I love LLMs. I agree with OP them expanding my hobby capacity as well. But I am constantly saying (in effect) “you sure…?” and tend to have a pretty good bs meter.

I’m still working to get my partner to that stage. They’re a little too happy to accept an answer without pushback or skepticism.

I think being ‘eager to accept an answer’ is the default mode of most people anyway. These tools are likely enabling faster disinformation consumption for the unaware.

Yes, you essentially have an impossibly well read junior engineer you can task with quick research questions like, "I'm trying to do x using lib y, can you figure that out for me." This is incredibly productive because in the answer is typically all the pieces you need but not always assembled right.

Getting the LLM to pull out well-known names of concepts is for me the skill you can't get anywhere else. You can describe a way to complete a task and ask for what it's called and you'll be heading down arxiv links right away. Like yes the algorithm to find the closest in edit distance and length needle string in a haystack is called Needleman–Wunsch, of course Claude, everyone knows that.

Once it gives me the names for the concepts I'm struggling with, I often end up finding the stackoverflow or documentation it's copy pasting.

I think so.

Junior devs can get plenty of value out of them too, if they have discipline in how they use them - as a learning tool, not as a replacement for thinking about projects.

Senior devs can get SO much more power from these things, because they can lean on many years of experience to help them evaluate if the tool is producing useful results - and to help them prompt it in the most effective way possible.

A junior engineer might not have the conceptual knowledge or vocabulary to say things like "write tests for this using pytest, include a fixture that starts the development server once before running all of the tests against it".

IMO experience provides better immunity for common hangups. Generated code tends to be directionally pretty good, but with lots of minor/esoteric failures. The experience to spot those fast and tidy them makes all the difference. Copilot helps me move 10x faster with tedious arduino stuff, but I can easily see where if I didn't have decent intuition around debugging and troubleshooting, there'd be almost zero traction since it'd be hard to clear that last 10% hurdle needed to even run the thing.

I wouldn't assume that at all. Most of the senior devs I talk to on a regular basis think commercial* LLMs are ridiculous and the AI hype is nonsensical.

* I put commercial there as a qualifier because there's some thought that in the future, very specifically-trained smaller models (open source) on particular technologies and corpuses (opt-in) might yield useful results without many of the ethical minefields we are currently dealing with.

It depends it think it’s less about how senior they are and how good they are at writing requirements, and knowing what directives should be explicitly stated and what can be safely inferred.

Basically if they are good at utilizing junior developers and interns or apprentices they probably will do well with an LLM assistant.

Ya. I think people that have better technical vocabulary and an understanding of what should be possible with code do better. That’s usually a senior engineer, but not always

It's the LLM paradox, seniors get more senior with them while juniors get more junior, creating a bimodal distribution in the future simply because juniors will start depending on them too much to learn how to code properly while seniors (who some may also exhibit the previous trait) will by and large be able to rapidly synthesize information from LLMs with their own understanding.

I had a couple of the most capable senior developers reach out to me to tell me how Github Copilot accelerated their productivity, which surprised me initially. So I think there's something to it.