The SecurityManager actually was a capability system! It just wasn't a pure capability system. Instead, capabilities enabled finer grained permissioning and optimization, which is IMO usually the right way to use them.

The problem with a pure caps system is that basically all software expects to have some ambient permissions. Changing that assumption not only breaks every API of every library, but reduces the usability by a lot too. For instance, it's commonplace for libraries to support tweaks to their behavior via environment variables or (in Java) system properties, but, those could also contain secrets, so that has to be a permission. In a pure capability design there's no way to grant a library the permissions it needs statically, instead a higher privileged component has to either do the read itself and pass the answer to the lower privileged part, or wrap its power into a capability object that exposes only what's needed. But then who writes that object? It's boilerplate so it'd make sense for it to come from the library itself, but, oh, that doesn't work because it's that code that's enforcing the security boundary. The user of the library has to write the code (or copy/paste and review it).

The SecurityManager design had some flaws but was basically a correct design for sandboxing most libraries (those that can't exfiltrate data obtained via side channel attacks):

1. You could assign libraries static sets of permissions, or not.

2. You could build strong capabilities using a mix of taking away a libraries reflection permissions, and AccessController.doPrivileged which terminated the intersection stack walk.

For instance, imagine a library that reads a config file (e.g. for logging configuration). There are two ways to handle this. You could change the library API so it requires a file capability. Java lets you do that - File[Channel] are capability objects. But, this is tedious for the developer. Now to use the library they have to write boilerplate to open the config file themselves so they can pass the opened capability in, which in turn means that basics like the location of the file can't be encapsulated in the library itself.

Alternatively you can just ship the library with a permissions file written in a simple DSL, that has a line like

    grant my-lib open ~/.config/my-lib/logging.json

Now the library API doesn't have to change, the developer doesn't have to open the file for the library, and the permissions can evolve alongside the evolution of the logging feature itself. The library user only has to review the permissions file when it changes, if they care about sandboxing, or ignore it if they don't.

This is much more usable! Capabilities in this model are still useful, but are the preserve of some special cases. For instance, complex permission checks can be done once and then encapsulated into a capability object (this is how file capabilities work, all the authorization logic is only on the open path and calling read or write is a much cheaper check). And of course APIs can be extended smoothly to support this, any time you have an API like:

    void setConfigFilePath(String path);

you add another method

    void setConfigFile(File file);

and make the first version delegate to the second. Now the library can have its permissions removed and be given a file capability instead, if that makes more sense.