Really excited to see this! Seeing some of the early comments here I think folks may not realize how awesome this would be in the server space.
After all, a big reason that NodeJS won a lot of popularity on the server is that, for many types of common webserver workloads (i.e. lots of IO, relatively minor CPU usage), NodeJS can actually scale much better than Java with its thread-per-request model.
With these virtual threads, though, you could get the best of all possible worlds - a webserver that scales like NodeJS, but without some of the "CPU starvation" issues you can hit in Node if one executing request doesn't yield, and also without having to worry about "function coloring" like you do in Node with async vs. non-async functions.
Really, really fantastic development, have been waiting to see when this would come out.
> For many types of common webserver workloads (i.e. lots of IO, relatively minor CPU usage), NodeJS can actually scale much better than Java with its thread-per-request model
Linux can handle a ginormous amount of threads quite well, would be interesting to see a deeper investigation to this theory.
The problem with doing it all native is that stack sizes are quite variable, especially in managed languages where modularity and code reuse works better, so it's common to have tons of libraries in a single project. The kernel won't object to lots of threads, but once those threads have been running for a while a lot of stack space will be paged in and used.
Loom solves this by moving stacks to and from the heap, where there's a compacting concurrent GC to clean up the unused space.
Yes. I am not as familiar with the underlying implementation of goroutines, but this description in the linked JEP sounds exactly how I understand goroutines to work:
> The JDK implements virtual threads by storing their state, including the stack, on the Java heap. Virtual threads are scheduled by a scheduler in the Java class libraries, whose worker threads mount virtual threads on their backs when the virtual threads are executing, thus becoming their carriers. When a virtual thread parks -- say, when it blocks on some I/O operation or a java.util.concurrent synchronization construct -- it suspends, and the virtual thread's carrier is free to run any other task. When a virtual thread is unparked -- say, by an I/O operation completing -- it is submitted to the scheduler, which, when available, will mount and resume the virtual thread on some carrier thread, not necessarily the same one it ran on previously. In this way, when a virtual thread performs a blocking operation, instead of parking an OS thread, it is suspended by the JVM and another one scheduled in its place, all without blocking any OS threads (see the Limitations section).
Yes. But, since there are _two_ kinds of threads in Java (os and virtual), you still have to be very careful never to block a virtual thread. In Go/JavaScript/Beam, it doesn't matter because you literally can't block a thread (while idle). This is the kind of thing that's not terribly useful until nearly every library you interact with is using it as well.
Also, there's no new syntax, so you're stuck with all the same thread pool concurrency we've been using for decades.
EDIT: It looks like I'm wrong about this:
> My understanding is that you won't have to worry about blocking a virtual thread, because all IO APIs are being modified to park when executed in the context of a virtual thread.
My understanding is that you won't have to worry about blocking a virtual thread, because all IO APIs are being modified to park when executed in the context of a virtual thread.
That said, you'd still need to worry about unsafe code, like JNA/JNI or other such thing that could still block. And I'm not sure there will be a way to prevent long running CPU task from clogging up the virtual thread executor threads.
> My understanding is that you won't have to worry about blocking a virtual thread, because all IO APIs are being modified to park when executed in the context of a virtual thread.
And, from what I read in the original JEP, the underlying system thread pool (which all virtual threads float between as needed) will be expanded when a virtual thread gets pinned, so you don't have to worry about exhausting your pool. (If you pin too many threads, obviously you'll be consuming more OS resources than you may have expected, but that's a different problem.)
What do you mean by pin here? Do you mean that a blocking IO will block the thread, but it will also add one more thread to the virtual thread executor pool? So blocking won't starve your virtual threads?
That's right. From the linked JEP, under "Scheduler":
> Some blocking APIs temporarily pin the carrier thread, e.g.most file I/O operations. The implementations of these APIs will compensate for the pinning by temporarily expanding parallelism by means of the ForkJoinPool "managed blocker" mechanism. Consequentially, the number of carrier threads may temporarily exceed the number of available processors.
Just to clarify, though, most currently blocking IO operations will not pin the carrier thread, because most IO operations you make from a webserver are network calls (e.g. to another API or the database), and those network APIs have been modified to not pin. From just a bit further up in the JEP:
> The implementation of the networking APIs defined in the java.net and java.nio.channels API packages have been updated to work with virtual threads. An operation that blocks, e.g. establishing a network connection or reading from a socket, will release the underlying carrier thread to do other work.
In Go one can block the native thread via using API that use blocking OS calls, like Linux file IO. In this case Go runtime allocates more native threads to run other language threads.
That's case for virtual threads also. It uses ForkJoinPool.ManagedBlocker to add additional threads.
"File I/O is problematic. Internally, the JDK uses buffered I/O for files, which always reports available bytes even when a read will block. On Linux, we plan to use io_uring for asynchronous file I/O, and in the meantime we’re using the ForkJoinPool.ManagedBlocker mechanism to smooth over blocking file I/O operations by adding more OS threads to the worker pool when a worker is blocked."
I learned an hard lesson in the Borland ecosystem.
Always go with the platforms languages, and the IDEs from the platform owners, even if others are more shinny.
Long term it always pays off to be the turtle, as the platforms move into directions not forseen by the shinny objects, and 3rd party IDEs keep playing catching up with SDK features.
Eclipse and NetBeans do exist, and... ehhhhh. I used NetBeans for a long time; couldn't stand Eclipse; and these days I only use IntelliJ. But the others absolutely exist, and it'd be hard to say that Apache and the Eclipse Foundation aren't deeply embedded in the Java ecosystem.
Eclipse is fine. Especially from VsCode where it uses the Eclipse language server. It boots fast, and when you run it with a modern JVM and GC the memory usage is leagues lower than IntelliJ.
If you need multi-platform then coroutines is still your best bet. But many people don't use Kotlin in a multi-platform way, and lightweight threads will be an easier migration path (and more compatible with Java libraries if you cant avoid one) compared to coroutines.
Node.js doesn't create a thread per request; it's single-threaded with evented I/O. You can use node-cluster to start more than a single thread to saturate multi-core CPUs and load-balance HTTP requests across these, but that doesn't make it thread-per-request.
I think my high school English teacher would agree with you that the sentence is written awkwardly (I can see the 'awk' note, in red, on my paper right now :) ). Here's how I parsed it:
> a big reason that NodeJS won a lot of popularity on the server is that, for many types of common webserver workloads [...], NodeJS can actually scale much better than Java with ~~it's~~ [Java's] thread-per-request model.
Twisol was right - I was trying to imply strikethrough using the Markdown syntax in an attempt to depict the idea of replacing "its" with "Java's". It didn't work as well as I hoped. In my mind I can see more 'Awk' scribbles on my post, and looking at it I agree :)
Adding in the 's is 100% my mistake. I've been guilty of using "it's" as the possessive form for most of my life, but that changes today! :)
(Forgive my grammar nazism.) The possessive form of "it" is "its": "The dog wagged its tail". But for basically everything other than pronouns and plurals, the possessive form involves adding "apostrophe s". In recent years, many people have tried to apply this rule to "it". But the problem is that "it's" is understood to be a contraction of "it is" or "it has"; furthermore, "its" already exists as the standard possessive form.
One thing I say to people using "it's" is that by analogy, you also need to say: "He got he's skills. She missed she's ride. They have they's meeting."
Thank you a ton for posting this! I've been doing this for most/all of my life and it didn't really make sense till now. I've had people explain it before but it didn't really make sense. Here's what I got from what you wrote (please correct me if this is wrong / kinda off in some way)
For most words, the possessive form is "<word>'s"
For pronouns (including it) there are different rules. He becomes his, she goes to hers, it goes to its.
Also, words that already end in s don't get the " 's " treatment.
(Question - for words that end in "s", we put the apostrophe after the existing, ending 's', yes?)
Thanks again for posting this - viewing the possessive form of it as (yet another English language) exception to the normal rule of " 's " is really helpful.
> One thing I say to people using "it's" is that by analogy, you also need to say: "He got he's skills. She missed she's ride. They have they's meeting."
This is a great distillation of the intuition I've always had, but never quite verbalized.
Sorry, yes, my sentence was poorly written with the ambiguous antecedent. Most Java webservers use a thread-per-request model, which is why Node can usually scale to more concurrent requests.
I can't think of any framework that still does one thread per request. Normally there is a a queue of incoming requests and they then get dispatched on a thread pool as threads return to the pool.
The challenge is normally that if any of the threads in the pool, as part of processing a request, needs to itself make an IO call, it will block. Ideally you'd want to park the request processing, return the thread to the pool, pick up the next request, until the IO is done where then on the next thread available from the pool you'd resume that request instead of picking another one. This is what the virtual threads will make really easy I think.
Maybe not _all_ possible worlds. You still have original Threads for things that need an actual OS thread. Its not a solution for UI threading.
There will be code that needs a native thread or non-preemptive threading and shouldn't be run on a virtual thread. In that sense there is method coloring but its yet to be seen how common a problem that will be.
Library writers and frameworks will need to sort out patterns for how to call Runnables in a safe way.
Yes but you always need original/kernel threads, regardless of what approach to async you need. The concept of a thread and a stack is hard-wired into the CPU.
W.R.T. code that needs a native thread: at the moment there's only two types of such code. One is code that uses Java's synchronized statement. That's supposedly just a, ehm, small matter of programming to fix. The other is calling into non-JVM controlled code. That's fundamental and no approach to scalable concurrency can fix it, not CPS/async/await or anything else because it's a foreign compiler.
But fortunately the JVM has some really interesting tricks up its sleeve there. For instance you can compile your native code using LLVM and then execute the bitcode on the JVM. Well, OK, currently GraalVM doesn't support Loom but hopefully Graal will be upgraded to do so as Loom gets integrated into HotSpot. And when it does, you will be able to call into code written in C/C++/Objective-C/Rust as long as that code can be recompiled with your own toolchain and as long as you can tolerate it being JITCd, also whilst benefiting from Loom's scalability.
Why do you say CPS can't fix it? C# works around this by having a synchronization context and ways to bounce around contexts. In this way C# async/await is able to ensure code is run on a specific native thread. Is that not a fix?
The idea is you need to understand your workload and run tasks on schedulers meant for that workload. You'd make sure to move that work to a context for long running tasks.
Not unlike how you might use a non-virtual thread pool in Java.... but it seems wrong to imply that you no longer need to think about this stuff.
That’s not function coloring, it is up to the caller whether to start it in a virt thread or a real one. Function coloring is having two methods do the same thing differing only in name and signature (eg. there is a blocking sleep and a non-blocking one).
>it is up to the caller whether to start it in a virt thread or a real one.
Sorta kinda but not when you're working in a framework that will call your code or working in some library where the abstracted code is non-obvious or uneasy to configure.
Maybe its not function coloring, although I wouldn't know what else to call it and I think its quite similar. What would you call the problem?
You should try Quarkus. It is a production framework built by Redhat. It uses Java-GraalVM under the cover to compile your entire webapp to an executable (like golang does).
It's just as fast.
Java is the highest performance and most tuned VM there is. I think you're really thinking of java from a long time ago, if ur thinking this
If you use gargantuan Java frameworks, you'll use a lot of RAM. Just don't do that. With Spring Boot and similar frameworks, the RAM usage is really just very modest. I'll give you startup times, since I am not a believer in Quarkus and Graal. And I wouldn't use Java for a serverless function that needs to spin up and respond quickly. But for a typical (blue/green-deployed) application in my world, startup time is still only a few seconds, which is fine for many applications. And I am not settling for "fine", just saying that the startup time isn't a big consideration, against a lot of things the Java (or Spring, in my case) ecosystem offers me.
You are probably right. It's one of those things where I've not seen a need to jump aboard. I'm still being fearful of reflection going to break on me. Probably irrational fear, but fed by me not understanding how it wouldn't break. Which I should study up on. Which I don't, since I don't have the need. And here I am ... vicious circle.
To be fair there is the new Kotlin based wiring API which avoids that with the caveat you need to instantiate everything manually etc. Which is probably a decent tradeoff for some folks.
I thought so too. And wrote simple web service using Helidon SE. It eats 300+ MB of RAM. I spend some time trying to optimize GC and all that stuff. Similar node service would eat 30 MB of RAM.
May be Graal would save us all. Until then Java is beyond salvation.
Other than a very very niche usecase, I really don’t see how eating 300 MB of RAM is so problematic when we quite literally have servers with terabytes of RAM. Yeah java can be configured to run GC all the time and target <100M of ram, but it rather runs the GC only seldom (jvm is actually one of the most energy efficient runtimed languages out there!) and trades memory usage to throughput.
Because in the cloud you're paying hefty price for every MB of RAM. For example with Jelastic you have 128 MB per cloudlet. And it's 2x difference between 120 MB and 130 MB. And with dedicated servers I don't have terabytes of RAM, I have two server with 24 GB each.
And no, you can't configure Java to target <100 MB of RAM. I configured it with -Xmx64m and it still eats around 300 MB. Java just fat and you can't do nothing about it at this time.
After all, a big reason that NodeJS won a lot of popularity on the server is that, for many types of common webserver workloads (i.e. lots of IO, relatively minor CPU usage), NodeJS can actually scale much better than Java with its thread-per-request model.
With these virtual threads, though, you could get the best of all possible worlds - a webserver that scales like NodeJS, but without some of the "CPU starvation" issues you can hit in Node if one executing request doesn't yield, and also without having to worry about "function coloring" like you do in Node with async vs. non-async functions.
Really, really fantastic development, have been waiting to see when this would come out.