I do worry about that and that's why the most important thing is to work for the right manager. Keep on their good side, make them feel like you're an asset even if not sprinting every sprint, and I've found that they'll go to bat for you at review time.