I also haven't experienced this much. I use the lazy loading functionality for vim-plug, along with async where I can. My neovim starts up in only a few milliseconds and avoids loading most plugins until I open specific file types or use a specific command.