As a (mostly) layperson, this seems like it could be a very significant paper. What are the odds we see the next few years of machine learning models based on RWKV like we have seen with transformers since the attention is all you need paper?
This model seem to be good when input context is large in comparison to OpenAI, would just wait till if someone productize it. Most likely it wine be just one paper but many new and old ideas combined
Being a RNN there is another trick: caching a long prompt, because RNNs only look back one step while transformers see the whole sequence. So you can load your long context only once and reuse it many times.
