They did savings by re-implementing their services and attribute those savings to the new tool / programming language.
I wonder what the saving would look like if they chose another tool for the second / optimized system. I doubt it would differ much if they went with Go, Java or stayed with Python.
It shows that Python with Django is literally 40 times slower than the fastest framework. Python with uvicorn is 10 times slower.
The use of languages like Python and Ruby literally results in >10x the servers being used; which not only results in higher cost, but also greater electricity use, and pollution and carbon emissions if the grid where the data center is located uses fossil fuels.
Not to mention, dynamically-typed languages are truly horrible from a code readability point of view. Large code bases are IMO difficult to read and make sense of, hard to debug, and more prone to bugs, without static types. I'm aware that Elixir is dynamically-typed, but it (along with JS) is an exception in terms of speed. Most dynamically-typed languages are quite slow. Not only do dynamically-typed languages damage the environment as they're typically an order of magnitude slower, they also lower developer productivity, and damage the robustness and reliability of the software written in it. To be clear, I'm in favor of anything that increases productivity. If Kotlin were 10 times slower, I'd be happy to pay that price, since it is genuinely a great language to work with, is statically typed, and developers are more productive in it. I'm not sure how Elixir mitigates the downsides of dynamic typing (maybe lots of 'type checks' with pattern matching?), but it would definitely be super-nice if a well-designed (Kotlin or Haskell like?) statically-typed language targeting the BEAM existed...
Since you mentioned them I think it's worth telling people to take those benchmarks with a larger grain of salt. They've become such a pissing contest that I don't know if they can be called "real-world". There doesn't seem to be much scrutiny of the implementations.
Take some Rust frameworks: they write out pre-computed header strings to the output buffer, completely bypassing what the framework's documentation recommends. Examples are actix[1] and ntex[2]. No one would ever do this in real life.
Now I like Rust, and it'd likely be some of the fastest even without these shenanigans (Axum and may-http don't do that, I believe). But I don't know if other languages/frameworks have benchmarks implemented in the same non-idiomatic way just to look better.
Hmm, the pre-computed header strings is definitely interesting optimization, and does seem to be a bit non-idiomatic – but IMO, this is the sort of optimization the framework itself should try and do.
If first several bytes of the header are going to identical for many requests, a super-optimized framework would ideally memoize it, and write it directly out, as this benchmark is doing.
(And if the framework itself is doing it, then any user of the benchmark would just inherit that optimization, and not have to resort to non-idiomatic optimizations...)
Did you even ever see implementation behind techempower benchmarks? There's NOTHING realistic in them. Those applications literally hardcode static content length header values to be faster. They are pretty good show of how low you can get to squeeze out performance but not one sane person will write code like that.
> greater electricity use, and pollution and carbon emissions if the grid where the data center is located uses fossil fuels.
to
> dynamically-typed languages damage the environment as they're typically an order of magnitude slower
is quite a stretch.
Do dynamically-typed languages inherently damage the environment? Or is it the fossil fuels?
Not that the appeal to the environment matters, because later on we have this:
> If Kotlin were 10 times slower, I'd be happy to pay that price, since it is genuinely a great language to work with, is statically typed, and developers are more productive in it.
> Do dynamically-typed languages inherently damage the environment? Or is it the fossil fuels?
My opinion is that slow languages that use 10x the electricity, with no ROI for the 10x energy use is bad.
High energy use, even if it's clean energy, implies a higher environmental toll. If a country were solely using nuclear and solar, higher energy use results in (1) more nuclear reactors constructed, and (2) more solar panels built. The manufacture and construction of both has an environment cost. Of course, with fossil fuels, the damage to the environment is potentially a lot worse.
> Not that the appeal to the environment matters
I don't think higher energy use is inherently bad. If we can improve the quality of life for human beings, then IMO a higher energy use is justified. I don't really believe in degrading our quality of life to lower our energy use.
My problem with many popular slow languages is that they have a negative ROI for the higher electricity cost. In exchange for 10x the energy use, you have a language that results in less-readable code (a serious issue), that causes more bugs / less-reliable software, etc. We literally a get negative ROI in exchange for 10x the energy use. Which is absurd and illogical.
If Hindley–Milner type inference had been more prevalent in the 1990s, I have a feeling dynamically-typed languages would have never take off. We're moving back to static typing with mypy, TypeScript, etc., but I'm hoping we move away entirely soon from using slow languages for writing servers serving large numbers of users.
If we actually got something in exchange for it, it wouldn't bother me so much.
For the vast majority of projects this makes no difference. If you are at Facebook scale? Sure. But then, you do what they did, write a VM to speed things up.
Dynamic typing doesn't cost much more money on average and thankfully the cost of energy itself is a motivation for companies to do rewrites. If they are paying a lot in server costs and electricity then they do typically rewrite to reduce the amount of servers.
For companies they primarily need to worry about running at a profit and getting to market quickly which dynamic languages do extremely well, and the costs in electricity and carbon aren't very high when your scale is small.
> running at a profit and getting to market quickly which dynamic languages do extremely well
This (or similar variants of this) is an assertion that's commonly made about dynamically-typed languages, but I don't think they hold any water.
Less readable code (due to the lack of types) makes it a lot harder to add new features, and harder to debug code as well.
Several years ago, I briefly worked on a fairly large codebase at a startup that was written in a Ruby on the backend, and CoffeeScript on the front-end. There were only around 60,000 active users. Yet, the dynamic typing made adding new features, or fixing a bug a truly painful and fragile experience. Needlessly painful, and slow. It literally reduced developer velocity.
I think once you cross a few hundred lines, dynamic typing becomes a handicap rather than an advantage.
All of this doesn't even touch on the energy use. Which I'll admit is irrelevant to most companies. Server costs even for popular web/SaaS/etc tech companies are often a tiny tiny fraction of overall cost, with most of the company's annual operating cost being employee salaries. (As I had stated earlier, I don't mind a language being slower - if it actually provided any advantages–like improved developer productivity, in exchange for that slowness.)
The main argument here was about the energy costs. But it sounds as though you don't like dynamically typed languages. That's fine.
Facebook, twitter and plenty of of billion dollar businesses were built with dynamic languages and many would argue they may not have even existed if they were written in staticly typed languages due to the slower up front time expenditure.
I like staticly types languages for large projects but enjoy thr development speed of dynamic ones. If you don't like dynamic that's completely fine
The only thing I'll concede is that using a statically-typed language without a good type inference system, might slow people down a tiny bit.
The serious downsides of dynamic typing means you might still win out (in terms of developer productivity, code readability, code reliability, and ease of debugging) even if you use a language like Java instead of PHP.
Stack overflow is miniscule compared to Twitter or Facebook. You can just book at Basecamp if you want to pick a smaller company. They are Ruby on Rails and they have been consistently doing well.
Ultimately billion dollar companies have been built on dynamic languages. There is nothing stopping you from succeeding with dynamic languages. There just isn't. There are tradeoffs and these companies made them.
I agree that there isn't much necessarily stopping a person succeeding due to the choice of language (unless it's some esoteric or otherwise unrealistic language).
Yahoo used C++ for instance (it would not be my choice, even though it's weakly statically-typed).
But, yea, like you said--there are tradeoffs.
I think dynamically-typed languages has the allure of letting you build quickly initially, but the downsides of dynamic typing start hitting pretty soon afterwrd.
I write a lot of small scripts in Python. It certainly is easier to throw something quickly together, especially when the data model is amorphous, with dynamic typing. But that doesn't mean I'd use Python for a large project.
Reading higher in the thread, some Elixir folks are saying that the techempower benchmarks used the wrong settings (debug mode, etc) for their Elixir benchmark.
The fastest Elixir framework on the list, phoenix, is about as fast as uvicorn. (Both around 10 times slower than dragon.)
I was mostly responding directly to these 2 statements from my parent comment:
> They did savings by re-implementing their services and attribute those savings to the new tool / programming language.
> I wonder what the saving would look like if they chose another tool for the second / optimized system. I doubt it would differ much if they went with Go, Java or stayed with Python.
I think second syndrome is probably a significant factor, but I can also believe that ditching Python was also a significant factor.
EDIT: I’m being rate limited because I guess my comments are too spicy for the HN mods, but anyway I agree that there’s no reason other non-Python languages would fare much worse than Elixir.
There is a reason: the BEAM is almost not prone to huge GC pauses. Bigger load results in every actor responding very slightly slower. Nothing else.
Many other systems don't have this property. They fall down under pressure.
Gosh, a huge chunk of HN is always so dismissive. At least read up a bit beforehand, man. The criticisms should be informed and benefit the readers, not only express a generic skepticism.
The beam just garbage collects each actor separately, and it so happens that much of the time your actor has finished before a gc happened so you never see the cleanup.
The beam also has a spectacular failure mode: OOM whenever messages come in at a higher rate than they are processed. The lack of backpressure mechanisms mean a huge amount of beam language developers spend way too much time recreating their own way for dealing with this or pretend it is not a problem at all. This means too many libraries in the ecosystem behave totally differently under load.
I can see how you would think that, yes. In practice I haven't noticed it except in super rare cases where processes (actors) hold on to huge binaries / strings -- which is one of the weak points of BEAM's GC.
I've been bit by this. In reality you need to know your shit when it comes to tuning the Beam and GC to achieve decent performance under load without triggering OOM.
Frankly I am seeing that as a myth, you seem to have made up your mind some time ago or judged by 1-2 occasions.
I am on ElixirForum every day and worked with Elixir for 7 years and have never seen anyone "perpetuate myths". I've seen some people willing to "increase adoption" which was always met with resistance by the wider community -- we believe growth should be organic.
Pretty sad stance from you though, I have no idea why people get so ticked off when another programmer wants to tell them about a secret weapon.
If you are not willing to try it, that's fair. Say that. Claiming you know stuff about the ecosystem while a guy who is there every day is not seeing that at all comes across as... strange. Biased. And not arguing in good faith. :(
> If you are not willing to try it, that's fair. Say that. Claiming you know stuff about the ecosystem while a guy who is there every day comes across as... strange. Biased. And not arguing in good faith. :(
I am not willing to drag others, such as those that wrote the repos, into a technical discussion with people out to act as you are.
The guy you are responding to was completely calm and reasonable. Didn't say anything attacking or otherwise. I'm not sure why you are seemingly trying to cast him (and the Beam) in such a bad light, with seemingly no reason to back it up.
Both of you are afflicted by that logical fallacy of failing to understand that you not encountering a phenomenon does not mean it does not happen or is rare, it just means you didn't encounter it.
If you try telling people that did encounter that phenomenon that in practice they wouldn't/didn't then you shouldn't be surprised if they question why they started talking to you in the first place.
> Do you genuinely fail to see how your behaviour proves my point?
I genuinely see only one thing: I asked you to elaborate but you are convinced that I am pretending to discuss while I, again genuinely, actually did want to discuss.
You asserting something about me, a person whose mind you cannot read is confusing and quite aggressive, in a very uncalled-for manner too. But as I said already -- have it your way, I disengaged because it became apparent you are not interested in discussing. OK. It's your right.
What's not OK is you claiming that I am not interested in discussing however, and I maintain that I was interested in discussing.
> You cannot just hound people with demands because you don’t like what they are saying.
1. I am not "hounding" you for anything, I asked a question.
2. You are again assuming my motivation and I assert that you have gotten it wrong. You that I "disliked what you said" is a borderline personal attack and an off-topic. I was confused why you claimed what you did and wanted you to elaborate, to find out what made you think like that and if I can change your mind with a few anecdotes and some facts (that are hard to look up because they require scanning a forum; yet they are there and are visible to everyone who engages with the platform).
BTW, if you really have known anything at all about the Elixir ecosystem you would know that its creator, to this day, engages with users on ElixirForum and asks for their feedback on what they find lacking. That sort of engagement and genuine discussion spirit that you claim I (as a part of the Elixir community) don't have.
That alone invalidates your point entirely.
I am disengaging second and final time, let future readers decide for themselves.
Calling someone “biased” and “acting in bad faith” is a personal attack and violates this site’s rules. People get rate limited for far less on this site.
I'd argue taking things out of context and deliberately painting the commenter in a bad light is not a nice forum discourse.
I said, very plainly and visibly, that my parent commenter's unwillingness to back up negative claims COMES ACROSS as biased and ARGUING (NOT "acting") in bad faith.
Come on now, this stuff is not hard, the message is literally up there. Not sure why you had to editorialize it and thus misconstrue it?
As I plainly explained, right at the top, it is not theoretical. There is no point engaging people with evidence if they are so dismissive of basic facts.
But then that is also true here. Your claims about him not saying what he plainly did are just bizarre.
TBH at the load we had we got substantial savings by eventually replacing usage of gen_server as well, though that probably isn’t a good idea much of the time.
The OOMs were largely being caused by calls to and from other services (i.e. kafka) so the answer proved to be in controlling the rate at which things come in and out at the very edge.
From what I saw I got the impression the Beam devs assumed memory and CPU usage go together so a system that is under load memory wise would also be CPU wise, but this isn’t the case if your fan out and gather involves holding large* values on which the response is based, even if for tiny amounts of time.
EDIT: *large meaning "surprisingly small" if you're coming from other universes.
The underlying problem we had* was the rate work was being completed was lower than the rate requests were coming in which causes the mailboxes on the actors to grow indefinitely.
In golang the approximate equivalent is a buffered channel that would start blocking because it has run out, but the beam will just keep those messages piling on to the queue as fast as possible until OOM. This is obviously a philosophical tradeoff.
* I should qualify that each request here had a life in low double digit ms, but there were millions of them per second, and these were very big machines.
Why weren't your processes dropping messages? Also I think you can tell the VM to not allow the process to exceed a certain message size and trigger some sort of rate limiting or scaling out
Edit: huh. I could swear the VM had memory limit options. Guess not. Time to rewrite it in zig!
Yeah, I think that's the assumption people had been operating under.
That team would have thoroughly endorsed a zig rewrite! It was a very odd situation where most of us liked erlang the language but found the beam to be an annoying beast, whereas most of the world seems to be the opposite.
That does not in any way explain a drop of 95%, which IMO is ridiculous and points to other issues.
The system they created now is totally different from the one they had. It’s more efficient by an insane margin. Choice of language seems like it would have trouble breaking the top 5 major reasons.
I migrated Rails apps to Elixir before, we reduced from 15 servers to 3, and 1 was basically "if crap hits the fan", we could have gotten away with 2 easily.
It's worrying that a supposedly high-quality forum like HN receives comments with no substance. If you have an actual counter-argument, let's discuss. If not, well, not an interesting exchange.
You keep insisting HN is not providing comments up to your standards. Let’s not go there OK? I just disagree with your analysis. I think it is too simplistic to state GCs cause this and Elixir somehow magically causes 95% efficiency boosts.
All I’m saying is that if you can drop 1330 servers just like that, there might be something more going on than Python’s slowness.
This is from experience. I have seen people create slow and fast systems with just about any tech. I can make Elixir crawl, I can assure you of that.
I have seen Python apps use 10 servers and reduced it to one as well. Same tech, just a more efficient mindset. It’s IMO a bit too simplistic to say systems with GCs fall over when under load.
Sure, if you want to expand the discussion to "everyone can make every tech stack act badly" then you might have had an argument. I don't find that argument compelling however -- it's borderline meaningless.
Also nobody used the word "magically" before you did. Note that.
What's your argument exactly? That Elixir is overrated? Or something else?
Furthermore, I am not insisting on my standards of the quality of comments. I am under the impression that's the expected quality of comments on HN at large.
Thank you for recognizing that we were going nowhere. Apologies if my tone was sharp.
I am not evangelizing tech -- I am a polyglot and I use what I find is best suited for a job, and Elixir happens to cover quite a lot of ground. That's all really. I also use Rust and Golang quite a bit.
I simply get ticked off when people start demeaning something without seriously working with it or even reading a bit beforehand. Sorry if I mistakenly put you in that group.
You’re comparing Elixir to Python and Rails. Many of us have seen Python replaced with other languages for an astronomical improvement. Python and Ruby are the slowest category of languages; they’re easily beaten and you need to offer some evidence as to why the improvement was derived from migrating to Elixir specifically rather than moving away from Python/Rails.
I'm not dismissive of Elixir, I'm dismissive that Elixir magically solved this problem in a way other languages couldn't. If you have some supporting evidence or rationale as to why Elixir is uniquely able to solve this problem, I'm happy to hear it, but so far you've offered up "low latency GC" which isn't unique to Elixir and itself doesn't adequately explain the degree of improvement over Python (GC latency alone doesn't reduce from 200 servers to 4). Again, I'm happy to entertain arguments about why Elixir is uniquely able to improve performance, but I'm not going to take it on faith (which you interpret as 'dismissive of Elixir').
Okay, how about "a runtime that has been extremely carefully crafted for the lowest possible latency all the way to the point of the hardware falling over"?
It's very hard to provide evidence unless we make a screen-share call where I show you real time dashboards of services being bombarded with thousands of requests per second and for you to see for yourself how the median latency numbers climb from 25ms to 45ms and then fall back to 20-30ms after the burst load subsides.
I find it difficult to just describe this because as much as I've seen it many times in practice, it's also practically impossible (NDAs and compliance nightmares) to demonstrate it to a programmer outside the companies I've worked with without violating all sorts of laws. :(
But yes, basically: a super latency optimized runtime, a GC that's not very sophisticated but it elegantly dodges most GC problems by simply releasing all memory linked to an actor as soon as it quits (and Erlang/Elixir encourage you to spawn many of those in the right conditions; not for every single thing though), and one of the very fastest dynamic languages in the world, probably second only to JS's V8.
All of that is combined with me working with several other programming languages and their hosting solutions which were tripping over themselves when 1000 req/s started coming in (looking at you, Ruby 3.X and Puma and a few other servers; or PHP, or Python).
TL;DR: reliability is much better, latency is predictable.
Weirdest thing is: people don't believe it. If you only knew the CTOs I worked with: they were extremely data-driven and they would not allow me or anyone to just pull all of that out of their bottom. All had to be proven with numbers, and me and my teams did that, many times.
I understand the skepticism somewhat, but you and a few others seem to look at Elixir through the lenses of "too good to be true", and IMO you should try relaxing that skepticism to some extent. And try to be little more sympathetic because again, I literally cannot give you the hard cold data without violating at least three laws.
Go also doesn’t have huge GC pauses (and moreover idiomatic Go generates very little garbage)and I have a hard time seeing how GC pauses would contribute so significantly. Java also allegedly has a very low latency collector.
Apologies, I was only responding to a single point which was meant to counter another. I am well aware that Golang's GC is world-class and it's my second most loved language after Elixir.
If and that's a very, very, very big if, the current open GC are really leading to to much pauses. Then you can go to Azul and buy a better VM and GC, further improving the performance compared to BEAM.
Yep, agreed, I am just listing possibilities. More often than not a performance loss in Erlang/Elixir is caused by GC pauses but you can do a lot to reduce or outright eliminate those.
You need to advance a theory about how the BEAM could make a 95% improvement in a way the alternatives cannot.
For example, a Go has a very good asynchronous story as well and while it may not be exactly as good as BEAM languages, it makes up for it in straight line execution performance.
I’ve personally rewritten a few carefully optimized Python applications in pretty naive Go (without significant rearchitecting) and witnessed 50X-1000X performance improvements. And moreover Go allows for more optimization beyond what Python allows (e.g., consolidating allocations and lifting them out of the hot path).
While you are probably right it's also true that python is just straight up slow - especially in normal configurations (django, flask, uwsgi, lambdas, etc) and elixir is pretty fast while offering a great/fun/friendly dev experience + BEAM-HA/scaling-benefits.
They could have also blown everything out of the water with C++ - or probably even golang - but if elixir can do it on 2-4 boxes it's fast enough.
Python is 100x slower than a real language (I like python and use it often, not meant as a dig at the language just stating facts)
My favorite thing about Python is writing prototypes. The big risk with writing a prototype is that it survives into production. Using Python ensures that the code will be replaced by real code
Python is great for other non-production code like Jupyter notebooks, numpy experiments, etc
How is calling Python "not a real language" just stating facts?
I'm not sure by which metric we are judging whether a language is real or not, but I'm fairly sure that almost anything anyone comes up with will include Python, considering it's one of the most used languages in the world at this point, and used for a fairly large variety of use cases.
Typescript/JavaScript has many of these properties except for GIL and maybe not typically interpreted. That means not a real language? Ruby also has a GIL.
I don’t even know what “proper package management” means in this context. Certainly there’s an official package management system and modules. C++ doesn’t have the latter (very fragmented user efforts and not commonly used in my experience) and barely the former (no adoption). So that means C++ isn’t a real language?
Python does have type checking by the way. Same as TypeScript - you annotate your types and you can run a program to verify your annotations. This is basically how TypeScript works although the typing in the latter is more mature by way of @types packages to let the community supplement adding typing information to third party packages. There’s no relation between the two so it’s not sound (in both cases), but in practice it’s quite useful.
Easy to learn hard to master syndrome. Your comment says nothing.
* GIL - Your use case more than likely reimplements one wheel or another. Whatever compute you're doing should be deferred to the right tool (DB, queue, etc). Otherwise, if it's I/O, you're on par with Go and NodeJS (See FastAPI 3rd party quarterly benchmarks as an example)
* No sound typing - I don't know what this means. If you're concerned with typing, you can use pydantic for highly performant type checking and input/output validation.
* No proper package management - Poetry is excellent, been around a long time and is the unwritten defacto tool. Pipenv imo is close second. This argument feels forced as no one argued Go isn't a language before gomod was solidified. The community was fragmented and people wrote/chose their own tool.
* Runtime type checking only - In a world of interprocess communication, I don't see how this is relevant. You're not using Pytyon to write firmware. If you're not writing tests and just depend on successful compilation, you're writing bad code. Tests not only cover your point but are an excellent self doc. An added value, imo, that isn't spoken about enough. Regardless, tests cover this.
* Interpreted - A good last point to nail in the "your comment says nothing". What does this imply? That it's easier to debug with an interpreter? That cold starts are slow and your design is flawed given the tools?
Anyway, with C and Go bindings, most arguments against Python fall short. It has its place, yes, but a much wider one than the bandwagon regurgitates.
They did savings by re-implementing their services and attribute those savings to the new tool / programming language.
I wonder what the saving would look like if they chose another tool for the second / optimized system. I doubt it would differ much if they went with Go, Java or stayed with Python.