Guessing based on what I see: The act of simply pulling out your phone can break the moment. Kids tend to zero in on it immediately, making it the new center of their attention.
I dunno, I have lots of photos of kids on my phone just like I described. It's true, sometimes they do do that, but also, because I take so many, they got over it.
