Seems they need to compare against "dumb" code completion. It seems that even when they are error-free, "large" AI-code-completions are just boilerplate that should be abstracted away in some functions rather than inserted into your code base.
On a related note, maybe they should measure number of code characters that can be REMOVED by AI rather than inserted!
boilerplate that should be abstracted away in some functions rather than inserted into your code base
Boilerplate is often tedious to write and just as often easy to read. Abstraction puts more cognitive load on the developer and sometimes this is not worth the impact on legibility.
Totally agree - LLMs remove the tedium of writing boilerplate code, which is often a better practice than abstracted code. But it takes years to experience to know what that's the case.
Say I'm faced with a choice right now -- repeat the same line twice with 2 minor differences which gets checked by IDE immediately, or create a code generator that generates all 3 lines which may not work well with the IDE and the build system which leads to more mistakes?
Agree with the first sentence. I used to tell people I never type anything more than periods and two or three letters because we already had good autocomplete.
On a related note, maybe they should measure number of code characters that can be REMOVED by AI rather than inserted!