> Especially interesting for software that are 99.9% of the time waiting for inference to come back to you.
In a different domain, I’ve seen a cli tool that requests an oauth token in Python be rewritten to rust and have a huge performance boost. The rust version had requested a token and presented it back to the app in a few milliseconds, but it took Python about five seconds just to load the modules the oauth vendor recommends.
That’s a huge performance boost, never mind how much simpler it is to distribute a compiled binary.
Python's startup cost is terrible. Same with Node. Go is very good, but Rust is excellent.
Even if a GC'ed language like Go is very fast at allocating/deallocating memory, Rust has no need to allocate/deallocate some amount of memory in the first place. The programmer gives the compiler the tools to optimize memory management, and machines are better at optimizing memory than humans. (Some kinds of optimizations anyway.)
TBH I'm still surprised how quickly Go programs start up given how much stuff is there in init() functions even in the standard library (e.g. unicode tables, etc)
I’ve spent some time optimizing Python performance in a web app and CLI, and yeah it absolutely sucks.
Module import cost is enormous, and while you can do lots of cute tricks to defer it from startup time in a long-running app because Python is highly dynamic, for one-time CLI operations that don’t run a daemon or something there’s just nothing you can do.
I really enjoy Python as a language and an ecosystem, and feel it very much has its place…which is absolutely not anywhere that performance matters.
EDIT: and there’s a viable alternative. Python is the ML language.
I know far too much about python packaging while only knowing a little about it.
I agree it’s hell. But I’ve not found many comprehensive packaging solutions that aren’t gnarly in some way.
IMHO the Python Packaging community have done an excellent job of producing tools to make packaging easy for folks, especially if you’re using GitHub actions. Check out: https://github.com/pypa/cibuildwheel
Pypa have an extensive list of GitHub actions for various use cases.
I think most of us end up in the “pure hell” because we read the docs on how to build a package instead of using the tools the experts created to hide the chaos. A bit like building a deb by hand is a lot harder than using the tools which do it for you.
That’s fair. I’m also thinking about the sheer size of Python apps that make use of the GPU. I have to imagine a C++ app performing neural network shenanigans would’t be >1GB before downloading weights.
In a different domain, I’ve seen a cli tool that requests an oauth token in Python be rewritten to rust and have a huge performance boost. The rust version had requested a token and presented it back to the app in a few milliseconds, but it took Python about five seconds just to load the modules the oauth vendor recommends.
That’s a huge performance boost, never mind how much simpler it is to distribute a compiled binary.