For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5;
For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2.
Llama 7B wasn't up to the task fyi, producing very poor translations.
I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on other vendors (even open source models).
I'm curious to see if others have gotten different results?
Yes, if you're just using Llama 2 off the shelf (without fine-tuning) I don't think there are a lot of workloads where it makes sense as a replacement for GPT-3.5. The one exception being for organizations where data security is non-negotiable and they really need to host on-prem. The calculus changes drastically though when you bring fine-tuning in, which lets a much smaller model outperform a larger one on many classes of task.
Also, it's worth noting that Replicate started out with a focus on image generation, and their current inference stack for LLMs is extremely inefficient. A significant fraction of the 100x cost difference you mentioned can be made up by using an optimized inference server like vLLM. Replicate knows about this and is working hard on improving their stack, it's just really early for all of us. :)
OpenAI aren't doing anything magic. We're optimizing Llama inference at the moment and it looks like we'll be able to roughly match GPT 3.5's price for Llama 2 70B.
Running a fine-tuned GPT-3.5 is surprisingly expensive. That's where using Llama makes a ton of sense. Once we’ve optimized inference, it’ll be much cheaper to run a fine-tuned Llama.
We're working on LLM Engine (https://llm-engine.scale.com) at Scale, which is our open source, self-hostable framework for open source LLM inference and fine-tuning. We have similar findings to Replicate: Llama 2 70B can be comparable to GPT 3.5 price, etc. Would be great to discuss this further!
> Llama 7B wasn't up to the task fyi, producing very poor translations.
From what I've read and personally experimented with, none of the Llama 2 models are well-suited to translation in particular (they were mainly trained on English data). Still, there are a number of tasks that they're really good at if fine-tuned correctly, such as classification and data extraction.
> I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on other vendors (even open source models).
I think you're definitely right about that, and in most cases just using GPT 3.5 for one-off tasks makes the most sense. I think when you get into production workflows that scale, that's when using a small fine-tuned models starts making more sense. You can drop the system prompt and get data in the format you'd expect it in, and train on GPT-4's output to sometimes get better accuracy than 3.5 would give you right off the bat. And keep in mind, while you can do the same thing with a fine-tuned 3.5 model, it's going to cost 8x the base 3.5 price per token.
Is that because translation is typically an encoder-decoder task and llama is decoder only or is there something else about it that makes the last difficult for llama?
Cost isn't the only incentive not to use an LLM service that resides in a foreign country. Around here, there are industries for which it's pretty much a no-brainer to avoid anything that sends data across the atlantic.
Although it wouldn't surprise me if today's Azure OpenAI offerings route to certain US-centric regions, I'd be very surprised if Azure isn't working day and night to try to provision OpenAI capacity everywhere they can in the world.
(Disclaimer: I work in the cloud organization at Microsoft, and these are totally my own thoughts and opinions and don't reflect any kind of inside knowledge I have. I think I can say that provisioning LLM capacity and GPU's is something we basically all have a tremendous amount of passion about.)
Let's say a French company would offer the same service in the US, swearing no data would be ever siphoned out of the US and no French intelligence service would be allowed to review the data. Would you be comfortable with your patient records being stored there or the business secrets of US companies?
Do you believe Microsoft can actually make the same promises and keep them? You don't have to answer the last question, of course, but please think about it. It doesn't matter where the LLM is located but who controls it and who holds the resulting data.
I don't think this is a promise Microsoft can make. The US Cloud Act states that Microsoft falls under US jurisdiction and it's legally bound to share foreign data if asked by US law enforcement.
"The CLOUD Act asserts that U.S. data and communication companies must provide stored data for a customer or subscriber on any server they own and operate when requested by warrant, but provides mechanisms for the companies or the courts to reject or challenge these if they believe the request violates the privacy rights of the foreign country the data is stored in."
I do think large tech companies do pretty well with customer data. As a former Googler I would be comfortable with my Gmail data residing in a foreign datacenter.
They do pretty well, except the Room_641A in the building which is allowed to do anything they what with production branch without it being visible to ordinary workers.
You can run 70B LLAMA on dual 4090s/3090s with quantization. Going with dual 3090s you can get a system that can run LLAMA 2 70B with 12K context for < $2K.
I built two such a systems after burning that much in a week on ChatGPT.
Have a client with many thousands of csv, json, xml files detailing insurance prices. Fundimentally they all contained the same data but wildly different formats because they were produced by different companies and teams. I used ChatGPT to deduce their format so I could normalize them. Easily underbid their current contractor who was using humans for the work and now I have an easy quarterly billing. :)
TBC, I probably could have optimized tokens but contract was profitable and time critical.
The one you could build for under 2K is last gen hardware.
* Chenbro Rackmount 4U Server Chassis RM42300-F (rack mount case Remove the air filter on 120mm fan. Put two decent 80mm exhaust at rear).
* Two used air cooled 3090s. About $650 a piece on ebay. Check slot width and make sure everything will fit on your motherboard. Do a burn in when you get them cause used GPUs can be hit or miss.
* 5950x CPU (overkill just had it)
* 128GB DDR4
* Motherboard with x570 chipset and dual pcie x16. These will birificate to x8 pcie 4.0 lanes to each GPU. This is enough bandwidth to push GPUs to max IME
* 1200W+ ATX power supply.
* ebay "u.2 pcie 3.84TB" and adaptor for m.2 NVME slot. (again what I had & it is cheap)
If you're going to really beat the thing I would power limit the 3090s to 320w (from 350w). Perf change is not really notable and keeps temps better.
From people hosting image generation models on Stable Horde I've heard that you can pretty severely underclock/undervolt your GPUs and keep them stable, massively reducing heat output and energy cost without losing nearly as much performance. I'm not sure if this transfers into text generation or not, this was from image generation workers that have a few seconds downtime between requests; however it might be worth a bit of research if you happen to be running consumer GPUs.
-----
From TheUnamusedFox, in August:
> 3090 down to ~260-270 watts (from 400) with minimal gen speed impact. Same with a 3080ti. It seems to be more stable with image generation than gaming, at least on my two cards. If I try to game or benchmark with this undervolt it is an instant crash.
From another user:
> this undervolting stuff is pretty sweet.
> undervolted_limits.png [1]
> max_power_limits.png [2]
> this is my before and after.
> a solid 200 watt drop for only 9.2% loss of performance
> not to mention the 30 degree drop in temps
>For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2.
You'll never get actual economics out of switching to open models without running your own hardware. That's the whole point. There's orders of magnitude difference in price, where a single V100/3090 instance can run llama2-70b inference for ~0.50$/hr.
No, they can't run it. llama 70 with 4 bit quantization takes ~50 GB VRAM for decent enough context size. You need A100, or 2-3 V100 or 4 3090 which all costs roughly roughly $3-5/h
Which model? I am talking about 70b as mentioned clearly. 70b 8b is 70GB just for the model itself. How much token/second are you getting with single 4090?
That is more than an order of magnitude better than my experience; I get around 2 t/s with similar hardware. I had also seen others reporting similar figures to mine so I assumed it was normal. Is there a secret to what you're doing?
TBH, Replicate is not a great way to run 7B beyond experimentation. You want a host with cheap consumer GPUs (like vast.ai) since the 4-bit requirements are so modest.
You either need a backend with good batching support (vLLM), or if you don't need much throughput, an extremely low end GPU or no GPU at all for exLlama/llama.cpp.
OpenAI benefits from quantization/batching, optimized kernels and very high utilization on their end, so the huge price gap vs a default HF Transformers instance is understandable. But even then, you are probably right about their aggressive pricing.
As for quality, you need a llama model finetunes on the target language (many already exist on Huggingface) and possibly custom grammar if your backend supports it.
I don't think translation is a great use case for ChatGPT and LLAMA. These models are overwhelmingly trained on English, and LLAMA2 which should have more data from other languages is still focused on languages w/ Latin/Cyrillic characters (so won't work well for Arabic, Hebrew, or CJK languages).
You're better off using models specialized in translation; General purpose LLMs are more useful when fine-tuning on specific tasks (some form of extraction, summarization, generative tasks, etc.), or for general chatbot-like uses.
I will say that GPT-4 is just incredibly expensive. For my app I only use it for advanced translations/corrections, and usually a combination of GPT-3.5+Wiktionary is able to get the more simple stuff done
Sure! I'm building a personalized AI language learning tutor using Open AI's API and ElevenLabs (for Text to Speech).
Right now it's basically a chat bot that you can use to practice conversing with. It provides corrections for the things you type. Eventually I'd like to try adding Whisper as well to allow users to speak out loud.
When you hover over a word, you get a translation. Initially I thought using Open AI for every word translation would be too much, but I've been able to get it down to ~36-40 tokens/request. (3-4 cents/1000 requests). I also began parsing and uploading some of this [Wiktionary data](https://kaikki.org/dictionary/rawdata.html) and am working on a feature that integrates the GPT-3.5 translation with this Wiktionary data.
A lot of these features are still in the works but you can feel free to try it if you like (https://trytutor.app).
What would be the best local standalone solution for translation model? Personal use, mostly self-education. 2 popular languages both ways (like en-spa, fr-ger). Free, pretrained off the github would be the best. I can try and train say 100M params LLM on 4090 RTX. But I'm not sure satisfactory result are achievable.
There are plenty of examples in the literature of using LLMs for translation beating the metrics of non-LLM models, even for languages for which there isn't a lot of data. Transliterating non-Latin characters helps a lot with accuracy as well.
what models would you use for translation? I am working on a language learning tutor (trytutor.app, very early) and GPT-3.5 turbo has been working fine, for the most part.
For foreign language corrections ("correct this German sentence and give a reason for the correction"), GPT-3.5 doesn't quite have the horsepower so I use GPT-4
Llama and GPT are auto-regressive decoder only architectures which for pure translation jobs are not the optimal architectures. Training seq2seq models or encoder/decoder models on datasets of sentence pairs designed for translation will likely allow you to use much smaller models. You will not be wasting parameters on general “language understanding” capability that Llama and GPT have if pure translation is all you need. T5 or Flan-T5 might be good starting points.
Yes, openAI is dumping the market with chat-gpt 3.5. Vulture capital behaviour at its finest, and I'm sure government regulations will definitely catch on to this in 20 or 30 years...
It's cheaper than the ELECTRICITY cost of running a llama-70 on your own M1.Max (very energy efficient chip) assuming free hardware.
I guess they are also getting a pretty good cache hit rate - there are only so many questions people ask at scale. But still, it's dumping.
Based on my research, GPT-3.5 is likely significantly smaller than 70B parameters, so it would make sense that it's cheaper to run. My guess is that OpenAI significantly overtrained GPT-3.5 to get as small a model as possible to optimize for inference. Also, Nvidia chips are way more efficient at inference than M1 Max. OpenAI also has the advantage of batching API calls which leads to better hardware utilization. I don't have definitive proof that they're not dumping, but economies of scale and optimization seem like better explanations to me.
I also do not have proof of anything here, but can't it be both?
They have lots of money now and the market lead. They want to keep the lead and some extra electricity and hardware costs are surely worth it for them, if it keeps the competition from getting traction.
gpt3.5 turbo is (mostly likely) Curie which is (most likely) 6.7b params. So, yeah, makes perfect sense that it can't compete with a 70b model on cost.
You think they are caching? Even though one of the parameters is temperature? Can of worms, and should be reflected in the pricing if true, don't get me started if they are charging per token for cached responses.
Together AI has new aggressive pricing where 70b are on par with gpt35 and everything smaller is fairly cheaper. The catch is the only 32k context length model as of today is their llama 7b which is fairly limited.
You (currently) need a GPU to run any of the useful models. I haven't really seen a business use-case that runs it on the user's computer, but given the hardware requirements it wouldn't be very feasible to expect.
So you'll have to figure out how to run/scale the model inference. Cloud GPU instances are generally very expensive, and once you start needing to horizontally scale it'll get messy fast.
At least at the moment it's expensive, especially if it's either very light usage or very intensive usage - you either need just a few seconds of compute occasionally, or lots of compute all the time requiring scaling.
The "lucky" ones in this scenario are small-medium businesses that can use one or a few cards on-site for their traffic. Even then when you take the cost of an A100 + maintaining it, etc. OpenAI's offering still looks attractive.
I know there's a few services that try to provide an api similar to what openai has, and some software to self orchestrate it, I'm curious how those compare...
> once you start needing to horizontally scale it'll get messy fast.
It gets expensive fast, but not messy, these things scale horizontally really well. All the state is encapsulated in the request, no replication, synchronisation, user data to worry about. I'd rather have the job of horizontally scaling llama2 than a relational database.
You can run the smaller Llama variants on consumer grade hardware, but people typically rent GPUs from the cloud to run the larger variants. It is possible to run even larger variants on a beefy workstation or gaming rig, but the performance on consumer hardware usually makes this impractical.
So the comparison would be the cost of renting a cloud GPU to run Llama vs querying ChatGPT.
>So the comparison would be the cost of renting a cloud GPU to run Llama vs querying ChatGPT.
Yes, and it doesn't even come close. Llama2-70b can run inference at 300+tokens/s on a single V100 instance at ~$0.50/hr. Anyone who can should be switching away from OpenAI right now.
> That's an exercise left to the reader for now, and is where your value/moat lies.
Hopefully more on-demand services enter the space. Currently where I am we don't have the resources for any type of self orchestration and our use case is so low/sporadic that we can't simply have a dedicated instance.
Last I saw the current services were rather expensive but I should recheck.
I bought an old server off ServerMonkey for like $700 with a stupid amount of RAM and CPUs and it runs Llama2-70b fine, if a little slowly. Good for experimenting
For use cases well within the capabilities of an LLM from last year, fine-tuned LLaMa 2 13B should/will blow ChatGPT out of the water: think "rate the sentiment of this text from 0-10".
I believe this because LLaMa-2 13B is more than good enough to handle what I call "quick search", i.e.
```
User: "What's the weather in Milwaukee?"
System: Here's some docs, answer concisely in one sentence.
AI: It's 73 degrees Farenheit.
```
YMMV on cost still, depends on cloud vendor, and my intuition agrees with yours: GPT-3.5 is priced low enough that there isn't a case where it makes sense to use another model. It strikes me now that's there's a good reason for that intuition: OpenAI's $/GPU hour is likely <= any other vendor's and inference time of LLaMa 2 ~= GPT.
I do think this will change with local LLMs. They've been way over-hyped for months, but after LLaMa 2, the challenges remaining are more sociological than technical.
For months now it's been one-off $LATEST_BUZZY_MODEL.c stunts that run on desktop.
The vast majority of the _actual_ usage and progress is coming from porn-y stuff, and the investment occurs in one-off stunts.
That split of effort, and lack of engineering rigor, is stunting progress overall.
Microsoft has LLaMa-2 ONNX available on GitHub[1]. There's budding but very small projects in different languages to wrap ONNX. Once there's a genuine cross-platform[2] ONNX wrapper that makes running LLaMa-2 easy, there will be a step change. It'll be "free"[3] to run your fine-tuned model that does as well as GPT-4.
It's not clear to me exactly when this will occur. It's "difficult" now, but only because the _actual usage_ in the local LLM community doesn't have a reason to invest in ONNX, and it's extremely intimidating to figure out how exactly to get LLaMa-2 running in ONNX. Microsoft kinda threw it up on GitHub and moved on, the sample code even still needs a PyTorch model. I see at least one very small company on HuggingFace that _may_ have figured out full ONNX.
Funnily enough, ONNX is getting a spike in mindshare over the last month in the _Stable Diffusion_ community. There's decent cross-pollination between local art and local LLMs, ex. LoRA's were first a thing for Stable Diffusion. So I'm hoping we see this sooner rather than later.
[2] Definition of cross-platform matters a ton here, what I mean is "I can import $ONNX_WRAPPER_LIB on iOS / Android / Mac / Windows and call Llama2.reply(String prompt, ...)"
[3] Runs on somebody else's computer, where "somebody else" is the user, instead of a cloud vendor.
My deepest thanks, I owe you one. Overlooked this completely. & spent dozens of hours learning way too much to still fall short of understanding how to make it work in ONNX.
For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2.
Llama 7B wasn't up to the task fyi, producing very poor translations.
I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on other vendors (even open source models).
I'm curious to see if others have gotten different results?