I imagine the rushing air can cool a rock pretty rapidly. The increase in terminal velocity will also tend to partially counteract the increased volume:surface-area ratio of larger rocks.