What languages and LSPs are you using? I've never experienced this with Python (pyright), Go (gopls) or Rust (rust-analyzer). I'm using tree-sitter for syntax highlighting. The only special thing is that I'm using lazy.nvim (the plugin manager) to load copilot.lua on the InsertEnter event.
I am using typescript with typescript-language-server. I'll give the lazy stuff a try, but I am not sure why this would make the editor more responsive outside of the initial load.