Unfortunately you got what you payed for! :) No one in the Linux world appears to be seriously investing in engineering a robust and reliable filesystem, with e.g. correctness proofs. We have only hobby projects.
At work, this all happened on a commercial Linux distribution which we do pay for. As far as I recall, their support was unable to resolve the issue, hence rebuilding all those VMs. I’m not on the server team, so I don’t know many details, but I was affected by this issue and it caused a lot of grief across the organization.
Are you sure brtfs is supported in production by your commercial Linux distribution? I would be surprised if this is true. RedHat and Ubuntu do not support it.
Facebook presumably uses xz in production too and that is a hobby project (as we all recently found out). My understanding is that development of Btrfs was not sponsored by any company and was entirely a "community effort". It certainly would explain why it's perpetually unfinished.
I honestly find it weird when I hear about companies like Facebook and Synology using it.
Facebook could easily work around failures, they've surely got every part of their infrastructure easily replaceable, and probably automated at some level. I'm sure they wouldn't tolerate excessive filesystem failures, but they definitely have the ability to deal with some level of it.
But Synology deploys thousands of devices to a wide variety of consumers in a wide variety of environments. What's their secret sauce to make BTRFS reliable that my work's commercial Linux distribution doesn't have? Surely there's more to it than just running it on top of md.
Maybe in the years since I was burned by it things have greatly improved. Once bitten, twice shy though - I don't want to lose my data, so I'm going to stick to things that haven't caused me data loss.
