The idea exists but it's clearly not true, except perhaps in specific, niche uses like re-rendering a continuously refreshed graph. Even for your image gallery, it makes for confusion. Back button does what? Shift-refresh does what? Just let the browser do what it does. If you want the images to render fast, use HTML 2.
Writing a web app with server side pages forces you to think about where the state lives. This is a beneficial discipline.
Google Maps would be the classic example of client-side refresh working so well that it's now the universal choice. At the time, it was a revelation, as the Mapquest-ish predecessors (if I recall) required a click and server-side refresh to scroll or zoom the map.
Of course the revelation here was that <a> tags weren't what we needed to move a map, but rather a click-and-drag plus scroll-wheel behavior to explore a huge image at various levels of detail. If the server-side page-by-page navigation paradigm is a lousy fit for information delivered over the internet, then it may make sense to re-invent the page load.
To use the language of a sibling comment, this brought things to a much more app-ish behavior. And eventually Internet maps have become, especially on mobile devices, an app. Hence the need to break server-side navigation may have foreshadowed the need to break out of the browser.
I think this can be generalized as: if you need to break out of "click and wait a moment and see a changed screen" paradigm, for something like a continuously scrolling map or a smoothly flowing server load graph, then you can make good use of client-side loading.
Exactly. But why a blog platform or news site or other content-focused website would feel the need to do that is beyond me. Not everything on the internet needs to be an app.
Writing a web app with server side pages forces you to think about where the state lives. This is a beneficial discipline.