Semi-related to this, but this type of statement and the reaction to it is how I discovered that I had basically assumed a manager role in all but title.

For the first little bit of my career, my coworkers and I would often joke about how the mistake one of us had made was going to get us fired. One day, I made a similar joke to another developer and was met with a look of sheer terror. Later that day, he pulled me into a room and asked me if I was serious. After doing my best to calm him down and assure him I didn't even have that type of authority, I came to realize that he viewed me as a manager even though I was technically at the same level in the org chart as he was (something that would be changed a few months later).

So that was the last time I ever made a joke about someone's job security and it made me realize that absent-minded statements from someone in a position of power are absolutely taken by others as things to be concerned about. I've been much more guarded with what I say to team members since that interaction, even those who aren't under me. It's not worth stressing your team over.