Symbolic story generation systems were capable of generating reasonably long and coherent stories. These systems derived much of their power from well-formed knowledge bases. But these knowledge bases had to be structured by hand, which limited what the systems could generate. When we shifted to neural networks, we gained the power of neural networks to acquire and make use of knowledge from corpora. Suddenly, we could build story generation systems that could generate a larger space of stories about a greater range of topics. But we also set aside a lot of what was known about the psychology of readers and the ability to reason over rich knowledge structures to achieve story coherence. Even increasing the size of neural language models has only delayed the inevitability of coherence collapse in stories generated by neural networks.
A primer such as this one makes it easier to remember the paths that were trodden previously in case we find opportunities to avoid throwing the baby out with the bath water. This is not to say that machine learning or neural network based approaches should not be pursued. If there was a step backward it was because doing so gave us a powerful new tool with the potential to take us further ahead. The exciting thing about working on automated story generation is that we genuinely don’t know the best path forward. There is a lot of room for new ideas.
My personal view is more critical than the author's. There was an approach that worked reasonably well but needed more work to improve it. It was abandoned for a flashy new approach that doesn't work and doesn't seem like it could be made to work ("Even increasing the size of neural language models has only delayed the inevitability of coherence collapse in stories generated by neural networks"). Why was the working approach abandoned? The answer seems to do with a certain tendency in academia to blindly follow new trends with the only criterion being how many published papers can come out of it. That is no way to make progress. That is the way to waste time.
Besides, this is not true:
>> There is a lot of room for new ideas.
Once your field has been taken over by neural networks, there is no room for new ideas. You are destined to tweak hyperparameters for the rest of your academic career.
I think this is a bit too cynical/negative. It's not true that neural net approaches don't work, they just have different strengths and weaknesses (strength being generalization/flexibility/ease of use, weakness being long term coherence, interpretability, reliability). There are plenty of examples of traditional methods being combined with neural net methods (eg https://ai.stanford.edu/blog/qagnn/ , https://arxiv.org/abs/1907.04799 , NeRF), and I think that is likely to happen in this context as well to some extent.
I really don't see generalisation, flexibility and ease of use (!) as strengths of neural networks and I'm shocked you think any of those are actually characteristics of neural networks.
To clarify I didn't mean that neural network approaches don't work. I meant that they don't work _for story generation_ certainly not if the goal is to generate coherent stories. In the article, the best examples of stories generated from neural network approaches (basically, nn-trained language models + x, where x is various clever tricks that try to patch up the gaping holes in neural nets' inability to generate a coherent story) look decidedly worse than the stories generated by symbolic approaches listed in earlier sections.
In my mind, the author of the article is being polite and trying to present the latest work in his field in a positive manner, possibly because of the natural tendency of researchers to not take a huge dump on each other's work. But some of the expressions he uses, like "in case we find opportunities to avoid throwing the baby out with the bath water" make it clear that he doesn't see the takeover of his field by neural networks as a step forward.
Btw, "cynical"? Should I take that as a personal comment? It sounds a little insulting.
After reading so many samples of machine generated text throughout the article, the conclusion paragraph felt entirely artificial and machine generated to me.
I’ve also been doing a lot of experimentation with gpt3 and gptj story generation recently. The problem is that although they always start off with something interesting, they will then begin to ramble, and finally go into an odd repetitive feedback loop. You can break this to some degree by injecting new, human generated content, into the prompt (which is what AI Dungeon does) but two problems always remain: the model has very little memory of context or previous events, and can’t plan into the future. So that while it will produce really interesting events that move the plot forward, it can only do so in response to an event that immediately precedes it. The “story” never finishes and there’s no real plot, it just meanders forward from event to event without development, long term consequences, goals or conclusion. This is because, as mentioned in the article, these models are always backwards looking and they don’t seem to build any kind internal model of the world they’re describing. And, the illusion of creativity is based on carefully constrained randomness, which, with enough brute force and luck, can produce interesting stuff, but can just as easily go off the rails.
There may be some way to carefully design prompts so that models learn how to both summarize all past events, which could then used as context for subsequent events. And also, to train them how produce coherent story summaries and then use those as a nucleus to develop properly structured stories - but we’ll need to investigate :)
I agree. One good way to enforce completed stories is to provide a single short story example ending in "The End." But that already makes each completion too expensive since it adds a substantial amount of text to each prompt.
Here are a couple stories that I would consider finished:
I'm also working a lot in gpt3/j story generation. One approach I've found that addresses some long-term coherence problems is to generate high-level, bulleted outlines upfront, then use those bullets (plus some metadata from previous/next bullets) as prompts to "expand" into longer, more-standalone scenes.
Of course, there's a lot of problems with this approach also, but it does help specifically with consistency across larger generated works.
Curious-- are you "invoking" the model for every user or caching some of the results?
I've been obsessed w/ using GPT3 for "personas" in conversations (for the agent to take on a character or follow instructions), but didn't seem feasible to make an API call (esp to Davinci) for every user
I agree the API is a bit expensive. I'm fortunate that this particular use-case is fit for caching.
It caches every result and only invokes OpenAPI upon clicking the "Generate More" button. The response is then saved for all future users of the site.
So far everything on the page has been generated for under $10. But I can see how it would become prohibitively expensive if generating unique text for each user.
There is a relatively new game called Wildermyth that does an amazing job at adaptive storytelling. Would love to see how it works. Maybe they use AI. Who knows!
One unexpected epicenter in culture wars was whether it’s ok to generate stories involving children. You can probably guess the stories people had a problem with.
Since models are becoming centralized, it’s a bigger deal than it seems. It’ll be interesting to see whether the AI community settles on one specific list of “don’t”’s, or if each company will have their own variant.
Symbolic story generation systems were capable of generating reasonably long and coherent stories. These systems derived much of their power from well-formed knowledge bases. But these knowledge bases had to be structured by hand, which limited what the systems could generate. When we shifted to neural networks, we gained the power of neural networks to acquire and make use of knowledge from corpora. Suddenly, we could build story generation systems that could generate a larger space of stories about a greater range of topics. But we also set aside a lot of what was known about the psychology of readers and the ability to reason over rich knowledge structures to achieve story coherence. Even increasing the size of neural language models has only delayed the inevitability of coherence collapse in stories generated by neural networks.
A primer such as this one makes it easier to remember the paths that were trodden previously in case we find opportunities to avoid throwing the baby out with the bath water. This is not to say that machine learning or neural network based approaches should not be pursued. If there was a step backward it was because doing so gave us a powerful new tool with the potential to take us further ahead. The exciting thing about working on automated story generation is that we genuinely don’t know the best path forward. There is a lot of room for new ideas.
My personal view is more critical than the author's. There was an approach that worked reasonably well but needed more work to improve it. It was abandoned for a flashy new approach that doesn't work and doesn't seem like it could be made to work ("Even increasing the size of neural language models has only delayed the inevitability of coherence collapse in stories generated by neural networks"). Why was the working approach abandoned? The answer seems to do with a certain tendency in academia to blindly follow new trends with the only criterion being how many published papers can come out of it. That is no way to make progress. That is the way to waste time.
Besides, this is not true:
>> There is a lot of room for new ideas.
Once your field has been taken over by neural networks, there is no room for new ideas. You are destined to tweak hyperparameters for the rest of your academic career.