You think about the implementation and how it can fail. If you don’t think about the implementation, or don’t understand the implementation, I would argue that you can earnestly try to test, but you won’t do a good job of it.
The issue of LLMs here is the proliferation of people not understanding the code they produce.
Having agents or LLMs review and understand and test code may be the future, but right now they’re quite bad at it, and that means that the parent comment is spot on; what I see right now is people producing AI content and pushing the burden of verification and understanding to other people.
> The issue of LLMs here is the proliferation of people not understanding the code they produce.
Let's ignore the code quality or code understanding: these juniors are opening PRs, according to the previous user, that simply do not meet the acceptance criteria for some desired behavior of the system.
This is a process, not tools issue.
I too have AI-native juniors (they learned to code along copilot or cursor or chatgpt) and they would never ever dare opening a PR that doesn't work or does not meet the requirements. They may miss some edge case? Sure, so do I. That's acceptable.
If OP's are, they have not been taught that they have to ask for feedback when their version of the system does what it needs to.
> pushing the burden of verification and understanding to other people.
Where was the burden prior to LLM's?
if a junior cannot prove his/her code as working and have an understanding, how was this "solved" before llm? Why can't the same methods work post-llm? Is it due to volume? If a junior produces _more_ code they don't understand, it doesn't give them the right to just skip PR/review and testing etc.
If they do, where's upper management's role here then? The senior should be bringing up this problem and work out a better process and get management buy-in.
>> If you don’t think about the implementation, or don’t understand the implementation, I would argue that you can earnestly try to test, but you won’t do a good job of it.
Previously the producers of the code were competent to test it independently.
This increasingly, to my personal observation, appears to no longer be the case.
They do test it, they just dont think about it deeply and so they do a shit job of testing it, and an incompetent job of writing tests for it.
Not by being lazy; smart diligent folk doing a bad job because they didn't actually understand what needed to be tested, and tested some irrelevant trivial happy path based on the requirements not the implementation.
Thats what LLMs give you.
Its not a process issue; its people earnestly thinking they've done a good job when they havent.
Testing is often very subtle. If you don't understand changes you made (or really didn't make because the LLM did them for you), you don't know how they can subtly break other functionality that also depends on it. Even before LLM's, this was a problem for juniors, as they would change some code, it would build, it would work on their feature, but it would break something else which was seemingly unrelated. Only if you understand what your code changes actually "touch", you know what to (manually or automatically) test.
This is of course especially significant in codebases that do not have strict typing (or any typing at all).
You think about the implementation and how it can fail. If you don’t think about the implementation, or don’t understand the implementation, I would argue that you can earnestly try to test, but you won’t do a good job of it.
The issue of LLMs here is the proliferation of people not understanding the code they produce.
Having agents or LLMs review and understand and test code may be the future, but right now they’re quite bad at it, and that means that the parent comment is spot on; what I see right now is people producing AI content and pushing the burden of verification and understanding to other people.