Thanks for the info! I thought I was joking about the possibility of using git-annex in our case, but you've made me realize that it's not out of the realm of possibility.
We could certainly shard our usage e.g. by customer - they're enterprise customers so there aren't that many of them. We wouldn't be putting the files themselves into git anyway - using a cloud storage backend would be fine.
We currently export directory listings to BigQuery to allow us to analyze usage and generate lists of items to delete. We used to use bucket versioning but found that made it harder to manage - we now manage versioning ourselves. git-annex could potentially help manage the versioning, at least, and could also provide an easier way to browse and do simple queries on the file listings.
We could certainly shard our usage e.g. by customer - they're enterprise customers so there aren't that many of them. We wouldn't be putting the files themselves into git anyway - using a cloud storage backend would be fine.
We currently export directory listings to BigQuery to allow us to analyze usage and generate lists of items to delete. We used to use bucket versioning but found that made it harder to manage - we now manage versioning ourselves. git-annex could potentially help manage the versioning, at least, and could also provide an easier way to browse and do simple queries on the file listings.