It starts at the first character, works forward one “token” at a time, and ends at the last character. Never moving back.
It feels like it knows where it’s going at the first character, even though it doesn’t.
It’s like it starts speaking a sentence and, by the time it’s done speaking, it’s written a syntactically correct Node.js application.
The way GPT communicates in English does seem similar to how humans communicate. The way GPT writes code doesn’t seem to come close to approximating how humans do - it’s an entirely different mechanism. Humans generally can’t write code without a cursor and backspace.
Speaking from ignorance (I've not studied attention nor transformers): This is my feeling too. I feel like the next step isn't far away: a more explicit model of the world with a mechanism to both query and "feed back on" that model, correcting mistakes.
If it's possible to get so far when that functionality seems in an important sense basically missing, imagine how far it'll go when that does happen.
It starts at the first character, works forward one “token” at a time, and ends at the last character. Never moving back.
It feels like it knows where it’s going at the first character, even though it doesn’t.
It’s like it starts speaking a sentence and, by the time it’s done speaking, it’s written a syntactically correct Node.js application.
The way GPT communicates in English does seem similar to how humans communicate. The way GPT writes code doesn’t seem to come close to approximating how humans do - it’s an entirely different mechanism. Humans generally can’t write code without a cursor and backspace.