The sudden lurch into paranormal book review at the end is interesting. I guess it's because it went down the path of, "How do we do it? How do we get there? Stay tuned and we’ll let you know." ... that last sentence is probably very likely to occur in conspiracy theory/paranormal/UFO texts.
It's very interesting that the model generated a basically coherent speech that could have come from any left-wing event or politician, given nothing more than "things are bad, what next" as a starting point. GPT-2 has correctly learned that Marxist thought is based on a form of catastrophism, as anyone who has read Marx will confirm.
It's going to be fascinating to see how people use this. My guess is "that sounds like an AI wrote it" will become an insult meaning predictable and content-free.
Even more fun will be putting the model into reverse and calculating a predictability score - if given the starting point of a real human written speech, GPT-2 rates each next word as highly likely, the overall speech can be said to be only N% insightful, where N is an actual scientifically defined measurement.
Many people seem to adopt dystopian catastrophism about AI but I feel somewhat optimistic. In the same way that automated spelling and grammar checkers can help people write better, a GPT-2 run in reverse could help people write clearer prose that gets to the point quicker, or perhaps even force people to accept when they don't really have anything new to say. If a speaker doesn't use it then someone in their audience will, after all.
Notice that "<|endoftext|>" delimiter. A lot of the samples I generated had that, and would then rapidly switch into a whole different tone or style. Maybe there was an error in their training where they somehow didn't separate training samples properly? I don't know enough about machine learning to say.
I also find it interesting that this sample got -4 points where the Sokal affair sample I posted got +4 points.
I imagine it has more to do with the emotions each sample evokes in various hackernews readers. Could it be that hackernews readers are likely to have a distaste for postcolonialism, but are likely to be fans of materialist rationalism? I think so, based on years of reading their comments :)
On the <|endoftext|>: GPT-2 and this model were trained by sampling fixed-length segments of text from a set of web pages. So if the sample happens to start near the end of one page then it will fill in the rest of the length with the beginning of another page. The model learns to do the same. TalkToTransformer.com hides this by not showing what comes after the <|endoftext|> token.
It's very interesting that the model generated a basically coherent speech that could have come from any left-wing event or politician, given nothing more than "things are bad, what next" as a starting point. GPT-2 has correctly learned that Marxist thought is based on a form of catastrophism, as anyone who has read Marx will confirm.
It's going to be fascinating to see how people use this. My guess is "that sounds like an AI wrote it" will become an insult meaning predictable and content-free.
Even more fun will be putting the model into reverse and calculating a predictability score - if given the starting point of a real human written speech, GPT-2 rates each next word as highly likely, the overall speech can be said to be only N% insightful, where N is an actual scientifically defined measurement.
Many people seem to adopt dystopian catastrophism about AI but I feel somewhat optimistic. In the same way that automated spelling and grammar checkers can help people write better, a GPT-2 run in reverse could help people write clearer prose that gets to the point quicker, or perhaps even force people to accept when they don't really have anything new to say. If a speaker doesn't use it then someone in their audience will, after all.