Yeah one of those mysteries of life. :/ One day we'll probably go back and try this on a few older versions to see if we can repro what happened. But yeah, it was right off the bat when we started our beta, 2 different people said they had to modify the yaml in order to get it to work (something we wanted to avoid altogether).
I don't know why it didn't work -- maybe they put the secret in the wrong namespace?