Using firefox too, I cannot replicate the behaviour. Could it be something from your side of things causing this ? Have you tried turning it off and on again ?
It seems from the code you first stretch it out to Infinity and then when scrollTop is bigger than scrollHeight, you go from negative Infinity back to 1. Negative values of scale() 'mirrors' the element. Maybe make sure scrollTop is never equal to or bigger than scrollHeight.
Like it's printed on some form of mathematical hyper-rubber sheet, and scrolling pulls it down until it stretches into invisibility, rendering the next rubber page visible.
It looks like the scrolling in Firefox is implemented in screen-height tiles, and when he's scrolling it's picking a smaller and smaller portion of the top tile (as the top portion of the tile is scrolling out of the window) but not updating the bottom coordinate of the rectangle it's supposed to render to (so instead of moving up, instead the smaller and smaller portion gets stretched out to fill the window). Then when the top tile is supposed to be completely off-screen, it jumps to the second tile in one go.