There are already some cool projects that help LLM go beyond the context window limitation and work with even larger codebases like https://github.com/jerryjliu/llama_index and https://github.com/hwchase17/langchain.

> Hate to break it to you and all the others.

Jeez. Their comment is quite obviously a complementary one in response to the limitation rather than a corrective one about the limitation.

The methods they employ are to improve the context being given to the model irrespective of the context length. Even when the context length improves these methods will be used to decrease the search space and resources required for a single task (think about stream search vs indexed search).

I’m also curious what paper you are referencing that finds that more context vs more relevant context yields better results?

A good survey of the methods for “Augmented Language Models” (CoT, etc.) is here: https://arxiv.org/pdf/2302.07842.pdf

Where can someone find and try ultra-long context length models?

Any links?

The longest one that is generally available is always going to be yourself :)

My context model is getting shorter and fuzzier.

… but still the weights are increasing ;)

There’s already project that help with going beyond the context window limitation like https://github.com/jerryjliu/llama_index

They also just tweeted this to showcase how it can work with multimodal data too: https://twitter.com/gpt_index/status/1635668512822956032?s=4...

Yea it's incredible. Looks like tooling in the LLM space is quickly following suit: https://twitter.com/gpt_index/status/1635668512822956032