>>> Well, the entire idea is to run locally, right? Now you have to worry about getting the runtime to work on 3 different OSs,
>llama.cpp
I'm well aware of llama.cpp, and run it myself. I'm also aware of the recent breaking change to the quantization format because 'everyone can just rebuild from the ggml'. And its difficulties with base level AVX or AVX512 (in fairness it focuses on M2 and has support for AVX2). And how you need to work to add additional non-Llama models to it (necessary due to the commercial clause). It's not quite ready yet. Even when it's ready, there's the work for actual deployment, and the mess of diverse user systems beyond what devs tend to run.
I'm not saying the challenge is impossible - but it's an extra hurdle for OSS.
>> I expect LLMs to be a ubiquitous UI paradigm.
> I don't expect that to be the case. LLMs are useful in text wrangling tasks, for which we have specialized software
Look at the recent photoshop generative fill feature for what could be done.
>what I do expect from an UI: a) that it is fully functional even when the machine is offline, and b) that it's responsiveness is independent from the load on someones datacenter.
Good ideas. I think for now BigCorp can avoid it by pushing LLMs as an addon UI and keeping the base UI functional. It's possible they'd ship smaller models on device too.
>Legislation is another good argument for open source LLMs taking over.
That I can buy. The legal details are however a bit beyond me right now, and I don't know what the current status is. It could push OSS or stop it.
>llama.cpp
I'm well aware of llama.cpp, and run it myself. I'm also aware of the recent breaking change to the quantization format because 'everyone can just rebuild from the ggml'. And its difficulties with base level AVX or AVX512 (in fairness it focuses on M2 and has support for AVX2). And how you need to work to add additional non-Llama models to it (necessary due to the commercial clause). It's not quite ready yet. Even when it's ready, there's the work for actual deployment, and the mess of diverse user systems beyond what devs tend to run.
I'm not saying the challenge is impossible - but it's an extra hurdle for OSS.
>> I expect LLMs to be a ubiquitous UI paradigm.
> I don't expect that to be the case. LLMs are useful in text wrangling tasks, for which we have specialized software
Look at the recent photoshop generative fill feature for what could be done.
>what I do expect from an UI: a) that it is fully functional even when the machine is offline, and b) that it's responsiveness is independent from the load on someones datacenter.
Good ideas. I think for now BigCorp can avoid it by pushing LLMs as an addon UI and keeping the base UI functional. It's possible they'd ship smaller models on device too.
>Legislation is another good argument for open source LLMs taking over.
That I can buy. The legal details are however a bit beyond me right now, and I don't know what the current status is. It could push OSS or stop it.