Before I buy, can you confirm the bridge is GDPR-compliant, AI-Act-ready, has a digital product passport, and passed its environmental impact assessment? Otherwise the local compliance officer will fine us before it even collapses.
>Before I buy, can you confirm the bridge is GDPR-compliant, AI-Act-ready, has a digital product passport, and passed its environmental impact assessment?
Great comment! We've added double-plus-good to your Palantir-Trumport account and 2% off your next Amazon purchase!
You might also consider a fast implementation of Elo and Bradley–Terry that I have been developing for some time: https://github.com/dustalov/evalica (Rust core, Python bindings, 100% test coverage, and nice API).
would you consider JS bindings? should be easy to vibe code given what you have. bonus points if it runs in the browser (eg export the wasm binary). thank you!
I am thinking about this for a while and I think I’ll vibecode them. Not sure about WASM, though, as the underlying libraries should support it, too, and I am not sure about all of them at the same time.
tldr; wikipedia iterates on <e^elo>, but that can go to zero or infinity. Iterating on <elo> stays between -4 and 4 in all of our observed pairwise matrices, so it's very well-bounded.
I am working on post-training and evaluation tasks mostly, and I built Evalica as a convenient tool for my own use cases. The computation is fast enough to not bother the user, but the library does not stand in my way during the analysis.
GPT-2 follows the very well-studied architecture of Transformer decoder, so the outcomes of this study might be applicable to the more complicated models.
That's what I'm worried about. What if we start using Rye, bake it into our projects and then Astral goes "Sike! Now we will collect all information available to the Rye binary and require 1$ for each build"
No. Native python ops in string suck in performance. String support is absolutely interesting and will enable abstractions for many NLP and LLM use cases without writing native C extensions.
It is absolutely true that there is massive amounts of room for performance improvements for Python strings and that performance is generally subpar due to implementation decisions/restrictions.
Strings are immutable, so no efficient truncation, concatenation, or modifications of any time, you're always reallocating.
There's no native support for a view of string, so operations like iteration over windows or ranges have to allocate or throw away all the string abstractions.
By nature of how the interpreter stores objects, Strings are always going to have an extra level of indirection compared to what you can do with a language like C.
Python strings have multiple potential underlying representations, and thus have some overhead for managing and dealing with those multiple representations without exposing those details to user code
Thank you! I excluded the coding tasks as most annotators don't possess this expertise. I trust them in comparing pairs of dissimilar model outputs that don't require any specific skill but commonsense reasoning.
The only manual analysis was when I checked the passed/failed prompts of the top-performing model.
It's all about timing and communities. In my echo chamber, everybody is currently making them a part of the product, but I'm interested in a more representative sample. Hiring usually precedes shipping.
I found code LLMs to be very useful for rewriting nested for-loops in Python as nice vectorized operations in NumPy. One has to be careful about unexpected array materialization, but generally it works really well and saves a lot of time.
