A UNIX-style process spawning interface, like exec() or posix_spawn(), requires the kernel to have some awareness of the executable format so that it can create a process from an executable image. It doesn't have to support every feature of the format, though.
A model in which this happens in userspace is conceivable, but it'd be pretty weird. Besides, it introduces a chicken-and-egg problem -- if you need a process to launch a process, you need some sort of special case for init.
IIRC, the binfmt elf code in the Linux kernel basically only does enough elf parsing to map everything into memory and kick off the linker. Then the linker does the rest, in userland. It matters because the binfmt code can be very hardened and minimal, whereas the userspace can go and do whatever it wants.
Of course the ELF parsing doesn't literally have to be done by the kernel, but there's no obvious good place to put it otherwise, considering it's a critical bit. If security is the real concern, then why wouldn't you just drop privileges while parsing ELF instead? I don't see the benefit to moving this code outside of the kernel. Maybe somehow moving it to a different protection ring would be helpful; but same for the rest of the binfmts.
I believe the Windows loader that loads PE files, is in fact in the kernel, along with a lot of other stuff. At the end of the day, when the Windows NT kernel was architected, speed was really important and security practices were not matured. I mean, the kernel handles or at once handled a lot of stuff, especially related to the GUI and even network RPCs and network services.
For a statically linked executable, the minimal parsing is enough to kick off the executable directly. After all, the linker is "just" a statically linked executable...
Then again, the kernel already has such a gigantic attack surface that it doesn't matter if the ELF parser is part of it.