The pipeline bottlenecks all changed in favor of bruteforcing the things that BSP had been solving with an elegant precomputed data structure - what BSP was extremely good at was eliminating overdraw and getting to where the scene could render exactly the number of pixels that were needed and no more. It's optimized around small, low-detail scenes that carefully manage occlusion.

More memory, bandwidth and cache means that more of your solutions are per-pixel instead of per-vertex and you can tolerate overdraw if it means you get to have higher polycount models. Likewise, the environment collision that was leveraged by the BSP process reduced the number of tests against walls, but introduced edge cases and hindered general-purpose physics features. Scaling physics leads in the direction of keeping the detailed collision tests at their original, per-poly detail, but doing things with sorting or tree structures to get a broadphase that filters the majority of tests against AABB or sphere bounds.

On a Wii(original Wii) 3D action game I helped ship, we just rendered the whole level at once, using only the most basic frustum culling technique; the hardware did the lifting, mostly through the z-buffer.

Adding to this, the nice thing about the bsp partitioning was you could also leverage it to make off screen monsters go to sleep or reduce their tick rate. Was helpful for optimizing AI as well as rendering. DOOM not only had some of the first pseudo 3d but also huge numbers of enemies... something that a lot of other games still cut down on

Read the quake pvs article linked to in this thread.

On top of my head as I remember it.. one of the reasons for Quake's use of a bsp was to allow back to front rendering of the world geometry without the use of a zbuffer. This was required to get decent performance with the software rasterizer.

I'm not 100% sure what's most commonly used these days, but for a large open world requiring data streaming, I could see the use for something like an octree and even portals.