I'm not saying that immorality alone will stop many people, but your rationale proves too much, and would stop you from intervening in any immoral act.
In a sense. The size of the group matters - inside the group you have trusted allies. Two foragers are safer than one; three are safer than two. And so one.
Most modern philosophers have taken the view that envy itself is immoral.