I can’t help shake the feeling that something closer to code (or specific training?) and further from natural language should be being used to configure these models at this stage of development. I was _astounded_ by the ‘Sydney document’ being MS’s way of ‘configuring’ New Bing.
Admittedly I have closer to a layperson’s understanding than an expert’s, but with some knowledge of how neural networks work, and having played moderately with ChatGPT, prompt engineering just seems _so unlikely_ to me to ever be able to create systems that behave as we desire them to behave anywhere close to 100% of the time, at least until the systems are orders of magnitude better at understanding (if that’s even possible).
There’s no other way to program it. There no “code” to speak of. The only way to control it is to give certain phrases more or less importance. You do that with direct prompts or tons and tons of training data.
This doesn't seem fundamental to the LLM paradigm. You can already tune some parameters in code after training time, eg temperature.
For example, you could imagine an LLM that as well as outputting probabilities for the next token, output the probability with which that token makes the response "offensive" or "helpful" or "playful". Then when it's time to use the model, you can slide some offensiveness and helpfulness parameters up and down depending on what the model is meant to do.
Perhaps this is a less powerful approach than training the generic model and telling it "Sydney is feeling particularly helpful today, and never espouses violence", but it's certainly an alternative. One problem is that experimenting with fundamentally different architectures for training GPT is very expensive, but experimenting using prompt engineering is relatively cheap.
I think what I’m saying is that that seems problematic (and I think the things we’ve seen from ChatGPT and Bing just emphasize this).
From chatting with these models, orders of magnitude better ‘understanding’* seems necessary before they’ll be able to actually reliably follow these ‘prompts’ during end-user conversations of the kind we’re expecting them to.
The prompts just can’t be precise enough, and the models can’t ‘understand’ them enough to extrapolate ‘spirit’ of the prompts as a human would (although, to be honest, I’m not sure a human could either because of the preciseness problem…).
This feels like a fundamental issue to me.
*I know - but if it looks like a duck and quacks like a duck - that’s been a controversial tenet of AI for decades…
They likely won’t get orders of magnitude better at understanding. But they will get way better at predicting what output we want. And as Edsger Dijkstra said, "The question of whether computers can think is like the question of whether submarines can swim."
A fleet of submarines designed and jointly made with the French is not as agile as a school of fish. What do the thought-streams of ten billion ChatGPT-X in simultaneous conversation look like in conceptual space? What do artists think?
Admittedly I have closer to a layperson’s understanding than an expert’s, but with some knowledge of how neural networks work, and having played moderately with ChatGPT, prompt engineering just seems _so unlikely_ to me to ever be able to create systems that behave as we desire them to behave anywhere close to 100% of the time, at least until the systems are orders of magnitude better at understanding (if that’s even possible).