As far as I know, muscle is only used when fat reserves are depleted or you don't eat enough protein for the brain's personal amino acid supply. I'm not particularly knowledgeable about this, though, maybe an expert can weigh in.
Not an expert, but from my own research muscle will always atrophy to its functionally base necessary mass, there are of course genetics involved in muscle mass retention as well that can’t be overlooked. But you can be in a caloric deficit and your body is burning fat(which often times this leads to lower energy and less intense resistance training) while still losing muscle mass. I don’t believe they’re mutually exclusive.
You’re 100% right though, the key to retaining muscle mass while in a caloric deficit is consuming sufficient protein, or even over-consuming protein. In starvation situations muscle is next up after fat when getting consumed for energy but it seems like for the most part the muscle mass deterioration during significant weight loss periods is a natural side effect of the process and lifestyle required to accomplish it.
>1) compared with persons with normal weight, those with obesity have more muscle mass but poor muscle quality; 2) diet-induced weight loss reduces muscle mass without adversely affecting muscle strength; 3) weight loss improves global physical function, most likely because of reduced fat mass; 4) high protein intake helps preserve lean body and muscle mass during weight loss but does not improve muscle strength and could have adverse effects on metabolic function; 5) both endurance- and resistance-type exercise help preserve muscle mass during weight loss, and resistance-type exercise also improves muscle strength. We therefore conclude that weight-loss therapy, including a hypocaloric diet with adequate (but not excessive) protein intake and increased physical activity (particularly resistance-type exercise), should be promoted to maintain muscle mass and improve muscle strength and physical function in persons with obesity.
The body needs protein to function (the body is partially built out of protein, and also proteins perform many important functions in the body), and it cannot be synthesized from fat or carbs. The body also has no store of protein. This means that outside of dietary protein, breaking down muscles is the only option.
Secondary to that, the body needs some amount of glucose. The best option for that is stored glycogen. When those stores run out the body can enter ketosis which reduces but does not eliminate the need for glucose. Fat consists of three fatty acids joined by a glycerol part. Most of the energy is in the fatty acids, but the glycerol can serve as a limited supply of glucose. However that is not really sufficient, so to get more glucose, and the other option is breaking down protein.
Any calorie deficit of 500 under BMR and the body will not just consume the fat stores but also muscle. That’s what I learned on the bodybuilding forums at least. Some people went too high on the deficit and ended up losing some muscle definition
It's more likely to happen if your deficit below TDEE is greater than 500 Calories. If you're 500 below BMR, you're almost definitionally losing weight at an unsustainably unhealthy rate, like much greater than 1% of your bodyweight lost per week.