That should be how this works, but unfortunately not. I have that toggle switched off, but I still regularly get this prompt.