macOS shows a modal with a list of offending applications and how much RAM they're using. Some of those applications are "paused", marked in red, some are "not responding", and some are responding normally.

You can pick applications from that list and choose between "Resume" (for "paused" applications) or "Force Quit".

At no point the system becomes unresponsive.