This is fascinating. It seems that GPT-3 does well with general, sensible questions that seemingly involve spitting out facts or short answers (prose, code), but does very poorly with tasks specialized programs could do.
I wonder what this means for the future of domains like automated theorem proving, logic programming where specialized searches are used everywhere, could there be a potential hybrid of that and more general language models such as GPT-3?
GPT-3 does very well with _analogies_. Those analogies don’t have to just be as simple as simple facts, but you do have to craft prompts to help guide it to the right analogy.
While the human is doing most of the work, this looks more like teaching another human than coding in a formally specified, machine-parseable programming language.
On one hand, this could decrease the barrier of entry to programming. On the other hand, it seems to leverage (and train) the same skills that are needed to express ideas clearly to humans, which is arguably much more applicable than learning a traditional programming language.
That's actually kinda cute, and oddly off putting at the same time.
I really am curious now in what happens back-end, in the model itself, and if it is actually learning. Really should follow Gwern and his circle, it's a lot better than the blind and baseless hype/criticism floating around the internet.
I wonder what this means for the future of domains like automated theorem proving, logic programming where specialized searches are used everywhere, could there be a potential hybrid of that and more general language models such as GPT-3?