How on earth have you had iTerm use up so much memory it's given you an error? I've been using iTerm for years on some absolute crap machines, abusing iTerm with tmux and numerous customizations, and have never received that error.
Unlimited scroll buffer, but good scrollback hygiene (I clear scrollback after I don't need it), and it ran out of memory by just sitting there — I was using something else during that time and all terminals had been at the shell prompts.
Such advice is not helpful. The infinite scroll buffer is useful for some people with long lived terminal sessions, and that helps them with their respective workflows and habits. I am one of them.
The parent ran out of memory with iTerm not because of the infinite scrolling buffer but because of iTerm having memory leaks. The memory leaks in iTerm used to be a big issue, and it appears they still have not fully fixed. Other terminal apps can swell up to over 1Gb of the RSS, yet they do not crash. Provided there is sufficient RAM available, the inifinte scrolling can prove to be benefecial.
I used iTerm on a 2011 iMac with 6GB of mismatched RAM until 2018 at work, and a crappy 2012 MBP with 4GB RAM (subsequently upgraded to 8GB) from 2012-2016 without issue. This is not normal at all lol
It's a 96 GiB of RAM machine, for chrissake! That ought to be enough for everybody!