There's a huge amount of technical jargon and sarcasm that makes it hard to see her point.
Basically she's saying that python async (which the current state of the art implementation uses libuv the same thing driving nodejs and consequently suffers from the same "problems") doesn't have actual concurrency. Computations block and concurrency only happens under a very specific case: IO. One computation can happen at a time with several IO calls in flight and context switching can only happen when an IO call in the computation occurs.
She fails to see why this is good:
Python async and nodejs do not need concurrency primitives like locks. You cannot have a deadlock happen under this model period. (note I'm not talking about python threading, I'm talking about async/await)
This pattern was designed for simple pipeline programming for webapps where the webapp just does some minor translations and authentication then offloads the actual processing to an external computation engine (usually known as a database). This is where the real processing meat happens but most programmers just deal with this stuff through an API (usually called SQL). It's good to not have to deal with locks, mutexes, deadlocks and race conditions in the webapp. This is a huge benefit in terms of managing complexity which she completely discounts.
> Python async and nodejs do not need concurrency primitives like locks. You cannot have a deadlock happen under this model period.
This is dangerously wrong and I would suggest that you reconsider the steps that got you to this understanding because something really important has been lost. It is absolutely critical to understand that deadlocks are not why you have locks. Correctness during concurrent operation is why you have locks. Deadlocks are a failure state when you do not have correctness during concurrent operation. So are things like double-increment and double-create.
Parallelism does not imply deadlocking, concurrency implies deadlocking, and both NodeJS and Python are concurrent runtime environments. And I can guarantee you that, with a little skull sweat, you can write a deadlock in NodeJS or Python. It is very easy. If you need some help, here's a trivial example (and ordinarily I wouldn't use a link shortener here but this one is hefty, it just goes to the Typescript playground): https://bit.ly/2Tvjyze
Also, as a concrete, real-world, yes-it-happens-here example of where locking is important, consider that I've recently built a dependency injection framework in NodeJS--tried to use others' first, but my situation isn't covered by existing ones--and had to resort to a mutex to avoid double creation of objects within a single lifecycle. Creation of objects within this lifecycle happens asynchronously--it has to, as the act of creating the objects might itself rely on asynchronous operations. So, if I were to have a diamond-dependency (A deps B and C, B and C dep D), I will non-deterministically, and based on the creation time of B and C, create either one or two instances of D. I rely upon a mutex, keyed upon the dependency being created, to ensure that this does not happen.
.
I would also submit that perhaps you should adopt a principle of charity and think real hard about whether your priors are correct before you start talking about what she "fails" to see. Rachel is one of those people who has Been Around and while I also have Been Around, I understand that Rachel has Been Around More and I probably should be listening more than I should be smarming at her.
>Parallelism does not imply deadlocking, concurrency implies deadlocking, and both NodeJS and Python are concurrent runtime environments. And I can guarantee you that, with a little skull sweat, you can write a deadlock in NodeJS or Python. It is very easy. If you need some help, here's a trivial example: https://codesandbox.io/s/2wxvp
Ok technically you're right. I am completely wrong when I say it can NEVER happen.
But let's be real here, you introduced DEADLOCKING deliberately by introducing LOCKS and by doing context switching at weird places to make it happen. When nodejs came out one of the selling points was the lack of deadlocks and locks.
Case in point: there are no lock libraries in standard NodeJS.
Think about it, why is a LOCK needed here? Let's say you didn't have locks AT all. Wherever the heck you are the current Node task technically has what is equivalent to a LOCK on everything. Why? because all node instructions are atomic and single threaded. This is what replaces LOCKS in nodejs. Your code example is just strange. The only place where your example is relevant is if there was another process.
>But as a concrete example of where locking is important, consider that I've recently built a dependency injection framework in NodeJS--tried to use others' first, but my situation isn't covered by existing ones--
Probably because, again, nobody really programs using DP in node let alone context switching and adding made up locks in the middle of all these injections and constructions. Whatever you're doing is probably very unique or (maybe, I don't know your specific situation) a sign over engineering. DP is a very bad pattern and is one of the primary sources of technical debt in code (especially when it's over 2 layers deep and in a diamond configuration)... but that's another topic. Anyway...
What is the point of "acquiring" a lock if in node I have a "LOCK" on everything? It makes no sense, whatever it is you're doing I am almost positive that there is a simpler way of doing it. Either way the dependency chain makes it obvious which needs to be created first and what can be created concurrently. The below psuedo code should produce what you're looking for without locks and with equivalent concurrency which is one of the main selling points of single threaded async.
b, c = await runAsync([B, () => C(await D())])
a = await A(b, c)
B is evaluated with D async, C is kickstarted after D, with B still being evaluated async. All of this blocks until both B and C are complete then A evaluates. Whatever the heck you're doing with locks, things should happen in the same order as the dependency chain in both my code and one with locks. There's really no other order these things can be evaluated. I would even argue that my code is indeed the canonical way to handle your diamond problem in node, no lock code needed as expressed by the standard node library.
Think about it, node includes high level functions for http but none for locks which are an even lower level concept than http. It must mean you aren't supposed to use locks in Node.
I will say you are technically right in the fact that a deadlock CAN happen. I was wrong in saying it can NEVER happen, but you have to realize that I have a point here. Your example is really going very very far out of the way to pull it off.
>I would also submit that perhaps you should adopt a principle of charity and think real hard about whether your priors are correct before you start talking about what she "fails" to see. Rachel is one of those people who has Been Around and while I also have Been Around, I understand that Rachel has Been Around More and I probably should be listening more than I should be smarming at her.
I was not smarming her, whatever smarming means, I am disagreeing with her just like you are disagreeing with me. There is NOTHING wrong with disagreeing with anybody. What is wrong is when you are proven wrong and you don't accept it. I accept that my statement of a deadlock NEVER happening in nodejs is categorically wrong.
"Being around" does not entitle you to anything. I hate it when people say this, nothing personal. Do you even know how long I've been around? Additionally, the overall main point of my post still stands, which you didn't even really address. I don't think Rachel gets the point of green threads. I think we can both agree I've made a strong point and maybe you should use your own charity principles on me.
NodeJS also doesn’t have a function to convert camel-case to PascalCase, should you not do that too because it’s not in the stdlib?
I’m going to be honest: you have not only not made a good point, you've gone out of your way to actively ignore that problems around concurrency regularly require one to use locks even in the absence of parallelism and have since long before multicore computers, and you're being weirdly hot-under-the-T-shirt besides.
Yeah well your post was rude and condescending. What did you expect with that attitude? Sure I'm angry, but there's nothing "weird" in my reaction given your rudeness.
>NodeJS also doesn’t have a function to convert camel-case to PascalCase, should you not do that too because it’s not in the stdlib?
This is entirely different from a Highly concurrent framework not containing lock primitives. A critical primitive is missing. It's like a math library missing the addition operator.
>you've actively ignored that problems around concurrency regularly require one to use locks even in the absence of parallelism and have since long before multicore computers
We're not talking about multicore/singlecore stuff. We're talking about NodeJS and Python Async Await and standard usage patterns.
There are other patterns that need locks but those are typically reserved for programming things like databases... something that a typical web programmer who writes NodeJS or Python doesn't deal with as web servers follow a stateless pattern that considers the usage of global state as bad practice.
> We're not talking about multicore/singlecore stuff.
If you write a python or nodejs handler, stateless or not, that does two subsequent async operations involving changes on shared resources, such as a database table, you need locks, because another request may come in while the first is in wait.
Perhaps you try to say that this is irrelevant when you allow only one request at a time, but that's extremely limited and not the scenario under discussion.
Nah I'm saying in the web application itself you shouldn't handle it. Python async/await or NodeJS.
You have to deal with deadlocks and race condition stuff in operations to the DB or any shared muteable state. That is obvious, but that is an external issue because web developers deal with shared muteable state as an external service that lives outside of python or nodejs code.
I mean if you count the transaction string or orm as part of dealing with locks within your web framework, then sure, I guess you're right? The locks on the DB are DB primitives though and not part of the web framework of language so I would argue that it's different. I guess reordering updates to happen on primitives in the same order could count as a web application change, but that's kind of weak as you're not really addressing the point:
NodeJS doesn't have locks, because you don't need locks in NodeJS and a deadlock should not occur unless you deliberately induce it.
The overall argument is the framework of NodeJS and python async/await is not designed for that kind of shared muteable state hence the lack of locks in nodejs std.
Also, never did I say a web developer doesn't need to understand or deal with concurrency. This is not true and I never claimed otherwise.
Additionally thank you for being respectful. (Take note
eropple)
This is a really important point. Exporting your locks to Postgres mean neither that they stop existing nor that you can’t wedge if it’s not written by clever programmers who better understand concurrency.
Yes it is an important point. Postgresql does not stop deadlocks or race conditions from happening. You deal with those in Postgresql.
But this isn't the topic of the conversation is it? The topic is locks and deadlocks in Python asyncio and NodeJS so ultimately irrelevant to your initial example of the amateur diamond dependency injection where normally no deadlock should be occuring regardless.
[I edited my post before his reply. Sorry for the confusion.]
Exporting your race conditions and washing your hands of them because the lock mechanism lives on the other end of a network socket rather than in your process space does not even rise to the level of “mere semantics”.
If you allow two NodeJS fibers to acquire remote locks—Redis redlocks, whatever—out of order by way of making asynchronous requests to it (noted only because you have a curious grip on that as being distinctive or meaningful here), you’ve still deadlocked and it is for all meaningful distinctions a deadlock of your processes (N >= 1). I state this only for completeness; there is no magic border at the edge of your process in which no, no, locks do not happen here. Locks control concurrency. When the problem set requires them, you use them.
I do not understand the spam of capital letters or the weird aggression. It’s like arguing about the coefficient of friction. The thing speaks for itself.
>I genuinely do not understand the spam of capital letters or the weird aggression about a trivial reality.
Then you better get with the program. Talking to people like the way you did won't make you any friends and will gain you many enemies. Don't worry though, I'm not that pissed off, just slightly miffed at your attitude. Also I like to use capital letters for emphasis. I guess you had a problem with that and decided to make it personal. Just a tip: don't act this way in real life, when you're older you'll understand.
>If you use Redlock to make a distributed lock over a Redis cluster and you allow two NodeJS fibers to acquire resources locks out of order by way of making asynchronous requests to it (noted only because you have a curious grip on that as being distinctive or meaningful here), you’ve still deadlocked and it is for all meaningful distinctions a deadlock of your processes (N >= 1).
Yeah because you're replacing your boolean in the earlier example with an isomorphic value. Either use a global js variable or a global value from redis. Same story. Nothing has changed from the locks you invented earlier.
Let me repeat my point. You shouldn't ever need to do the above in NodeJS because the area where the asyncio in python and NodeJS operate in are stateless web applications. That's why NodeJS doesn't have locks. You have to go out of your way to make it happen.
>There is no magic border at the edge of your process in which no, no, locks do not happen here. Locks control concurrency. When the problem set requires them, you use them.
And your point is? I don't understand your point. Clearly nothing I said was to the contrary.
Let's say your problem set is writing a database. Then locks makes sense. Does NodeJS make sense for this problem set? No. Do Locks make sense for NodeJS? No. Global mutable state is offloaded to external services and that is where the locks live. This is the trivial reality.
Let's stay on topic with reality. In what universe does your diamond dependency need a dependency injection framework with locks in nodejs? If you need locks your fibers are sharing global state and you've built it wrong.
b, c = await runAsync([B, () => C(await D())])
a = await A(b, c)
That you conflate “async IO” and promises may be why you’re in this hole in the first place.
Async IO uses promises to abstract out it’s select (or moral equivalent) but promises are not async IO. A weirdly prescriptive attempt at dictating what these are “for” don't do much to obscure the thing—they speak for themselves.
I’m still really confused why a callback-hell topographical sort and process would be somehow better than a cache, a lock, and a breadth-first search—not least because it’s easier to follow and is also, anecdotally, faster—but clearly these mysteries are just plain beyond my pay grade.
>That you conflate “async IO” and promises may be why you’re in this hole in the first place.
It's all just single threaded cooperative concurrency with context switching at IO. The isomorphic apis on top of this whether it's callbacks, async/await or promises is irrelevant to the topic at hand.
>I’m still really confused why a callback-hell topographical sort and process would be somehow better than a cache, a lock, and a breadth-first search—not least because it’s easier to follow and is also, anecdotally, faster—but clearly these mysteries are just plain beyond my pay grade.
I'm confused as to what the hell you're talking about. "callback-hell topographical sort and process" Wtf is that? Where were callbacks used in my example? Where was sort used?
Do you not understand that the dependencies determine the order of construction? That's it, it doesn't matter what technique you use the overall steps are the same. There is no bfs or callback hell going on. You manually instantiate the dependencies and choose what's async and what is sync. No need for locks.
Are you talking about something that takes a dependency graph and constructs the instance from that? If you want to do that your algorithm is incorrect. You need Post Order DFS, BFS won't work, but both BFS and DFS are O(N) so in terms of traversal over dependencies it's all the same.
class Node:
def __init__(self, createAnObject: AwaitableFunction[Any...], dependencies: List[Node])
self.deps = dependencies
self.constructor = createAnObject
async def constructObjectFromDependencyTree(root: Node) -> Any:
if root is None:
return None
else:
instantiatedDeps = await runAsync([constructObjectFromDependencyTree(node) for node in root.dependencies])
return await root.constructor(*[i for i in instantiatedDeps if i is not None])
The algorithm is bounded by O(N) where N is the amount of total dependencies.
If you want to construct an object with a total of N dependencies then no matter how you do it, the operation will ALSO be bounded by O(N). In terms of speed, it's all the same, but the above is how you're suppose to do it.
The above algorithm should give you what you want while providing concurrency and sequential execution exactly where needed. No callback hell, no promises, no sorting, no external shared state and no locks.
Regardless, if you're building Objects that necessitate such algorithms you are creating technical debt by creating things with long chains of dependencies. You should not be using your primitives to create large dependency trees; instead you should be composing your primitives into pipelines.
Additionally, relegating so much complexity to runtime is a code smell. If there aren't too many permutations bring it down to a manual construction with your code rather than an algorithm/framework.
Man, don’t do this here. You’re angry because I provided cites. You’re angry because you were overconfident, sweepingly general, wrong, and (worse) trivially proven wrong so you’re trying to well-actually out of it by being angry. I hasten to note that I am not being judgmental; I have been there. The best thing you can do is take the L and learn from it, dude. Everybody goofs sometimes, but you recall the best advice to take when you find yourself in a hole, yeah?
And...“typical web programmers”? I don’t know how relevant that is except in the light that typical web programmers use the tools built by folks who understand concurrency we’ll enough to build abstractions that make “you don’t need to think about concurrency“ mostly safe even though they’re completely wrong. Somebody’s gotta be doing that for you.
>Man, don’t do this here. You’re angry because I provided cites.
No I'm angry because of your attitude. What the hell is a cite?
I'll quote the drivel coming out of your mouth. Most of it is personal and doesn't even refer to the topic at hand:
>Quit while you're behind, my dude.
>This is dangerously wrong and I would suggest that you reconsider the steps that got you to this understanding because something really important has been lost.
>I would also submit that perhaps you should adopt a principle of charity and think real hard about whether your priors are correct before you start talking about what she "fails" to see. Rachel is one of those people who has Been Around and while I also have Been Around, I understand that Rachel has Been Around More and I probably should be listening more than I should be smarming at her.
>You’re angry because you were overconfident, sweepingly general, wrong, and (worse) trivially proven wrong so you’re trying to well-actually out of it by being angry.
>I hasten to note that I am not being judgmental; I have been there. The best thing you can do is take the L and learn from it, dude. Everybody goofs sometimes, but you recall the best advice to take when you find yourself in a hole, yeah?
Nothing I quoted was evidence that I'm wrong, but everything I quoted had a condescending attitude and very personal. This is not how you engage in respectful conversation. It wasn't my plan to "do this here." I don't really give a shit, I'm just saying you mouth off with that garbage of course the person on the other side is going to be a little pissed off. What the hell did you expect? My reaction isn't "weird" like you said earlier, it's normal to someone who is Rude. This was the small point I was trying to make which you expanded on with a very personal remark.
That being said I'm not that angry, just a little, this is the internet after all. I don't care.
Also did you not see my first post? Did you not see me admit to being wrong on something? I have no issue with doing that. This is not a problem for me. Ever. If I'm still arguing with you it means I disagree with you. No actually scratch that, a better way of saying it to you specifically is that it's not that I disagree with you, it's that you're flat out completely wrong and I'm right. See that? Same tact you have.
>And...“typical web programmers”? I don’t know how relevant that is except in the light that typical web programmers use the tools built by folks who understand concurrency we’ll enough to build abstractions that make “you don’t need to think about concurrency“ mostly safe even though they’re completely wrong. Somebody’s gotta be doing that for you.
web programmers need to understand concurrency. In external services like databases you still need to deal with locks, race conditions and deadlocks. These don't disappear, I never said that. The topic is Python and NodeJS and async await and that is the context I am referring to... please stay on topic.
I never said “you don’t need to think about concurrency“ <--- this right here was made up and a total lie.
The rest of that paragraph is incoherent. Somebody is doing <what> for me?
On a side note, you haven't given me any concrete examples of when you need to realistically use your made up locks in nodejs. Your last diamond dependency scenario and code sample made no sense from a practical standpoint.
Basically she's saying that python async (which the current state of the art implementation uses libuv the same thing driving nodejs and consequently suffers from the same "problems") doesn't have actual concurrency. Computations block and concurrency only happens under a very specific case: IO. One computation can happen at a time with several IO calls in flight and context switching can only happen when an IO call in the computation occurs.
She fails to see why this is good:
Python async and nodejs do not need concurrency primitives like locks. You cannot have a deadlock happen under this model period. (note I'm not talking about python threading, I'm talking about async/await)
This pattern was designed for simple pipeline programming for webapps where the webapp just does some minor translations and authentication then offloads the actual processing to an external computation engine (usually known as a database). This is where the real processing meat happens but most programmers just deal with this stuff through an API (usually called SQL). It's good to not have to deal with locks, mutexes, deadlocks and race conditions in the webapp. This is a huge benefit in terms of managing complexity which she completely discounts.