No, but when you choose NextJS you're choosing a product and should evaluate the reasons for doing it. It's abstracting away a substantial amount of pain points in deploying and optimising react apps. The OP even talks about some of the things he's losing out on by dropping it. We spend so much time as developers moving on to the next thing, instead of shipping and worrying about a few mins of build time when deploying (which is nothing).
It's not only a few mins when deploying. Often you want to test the production build locally and it feels like waiting forever. Especially since clicking around in an app for the dev build is slow during cold use. I think it has gotten worse as I don't remember build times taking so long in next versions 5 thru 9 or so.