The team at Heroku (where I currently PM) is constantly trying to improve our communication and documentation. We're definitely sorry that this caused problems, and we'll work even harder to make sure that our communication calls out any potential issues. Again - thanks for reaching out to us, and let us know if we can help.
Heroku did their best here. They reached out to us (twice) advising of changes and linking to a document that describe EXACTLY the bug that we discovered (later). Honestly, I don't feel bad about his bug because even if I would have read the alert to the letter, we would not have audited the entire codebase because we don't have that luxury of time. Yes, it took our whole operation down...but we found it, fixed it and now we're back up. ...it's all in the game. -@eatabit
That's all true but once you start accepting illegal input on a protocol for a long enough time you can't just suddenly go and break things without an automated alert to the customer when that particular thing starts acting up.
After all it would not be that hard to scan for which customers are going to be bitten by that particular change when it actually happens rather than using some fire-and-forget email.
