Isn't equating active parameters with cost a little unfair since you still need ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

brokensegue on April 17, 2024 | parent | context | favorite | on: Mixtral 8x22B

Isn't equating active parameters with cost a little unfair since you still need full memory for all the inactive parameters?

tartrate on April 17, 2024 | [–]

Well, since it affects inference speed it means you can handle more in less time, needing less concurrency.

sa-code on April 17, 2024 | [–]

Fewer parameters at inference time makes a massive difference in cost for batch jobs, assuming vram usage is the same

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact