I think it comes down to a mix of (a) how people (and robots) organize and find information (b) tool limitations and genericity.
Regarding (a),I personally like the view that the content lives in a "flat world" on top of which we collate different structures to organize/filter a same set of contents. In that worldview, web users entry-points can be more than directory-listings. A great inspiration is how Wikipedia offers a way to find which articles use a given picture: each picture acts like a "category", the same way that "recent-changes" is another filter on the same "flat world" of articles.
However, what is immensely difficult is to standardize these in a world where de-facto implementations burgeon, flourish, and eventually becoming out of touch (e.g., site-maps, RSS, OpenGraph). Hence we are stuck with very limited but very generic tools (b) for which rules like "directory-listing on the slash-separator" or "generate a JSON of the whole site connections to display as an interactive graph" (which I do on my personal blog) merely are local work around which require a bit of duck-tape to work.
Regarding (a),I personally like the view that the content lives in a "flat world" on top of which we collate different structures to organize/filter a same set of contents. In that worldview, web users entry-points can be more than directory-listings. A great inspiration is how Wikipedia offers a way to find which articles use a given picture: each picture acts like a "category", the same way that "recent-changes" is another filter on the same "flat world" of articles.
However, what is immensely difficult is to standardize these in a world where de-facto implementations burgeon, flourish, and eventually becoming out of touch (e.g., site-maps, RSS, OpenGraph). Hence we are stuck with very limited but very generic tools (b) for which rules like "directory-listing on the slash-separator" or "generate a JSON of the whole site connections to display as an interactive graph" (which I do on my personal blog) merely are local work around which require a bit of duck-tape to work.