This is probably a simple question, but it may not be obvious to others as well: How effective are caches on a system that is running multiple different processes at one time, much less hundreds? Is each working set that a process uses at any certain time small enough to fit in the cache along with the others?
Looking back at caches that used to be 4k or 64k (external) and are now 256k and measured in megabytes (external), I suppose that could be the case.
The number of threads a CPU can run at once is equal to the number of cores (we'll ignore hyperthreading for the sake of simplicity), and the number of cores is equal to the number of (L1) caches.
If you're referring to kernel-level multitasking, then these processes don't really run concurrently from the CPUs perspective; in fact, they're switching interval is quite long (relative to the processor speed).
However, the cost of bringing in a thread's data into the cache once it's been scheduled by the kernel is not negligible at all, and is part of the significant total task-switching cost.
Looking back at caches that used to be 4k or 64k (external) and are now 256k and measured in megabytes (external), I suppose that could be the case.