Okay, here's some feedback: you mention that you're using "real" vim, yet refer to the file browser controls as that "hjkl" thing. No app that isn't vim or isn't trying to partially resemble vim has used that obsolete "uh, down means up, so left means down" hjkl arrowkey arrangement for decades. Everyone else, all apps, spreadsheets, games, other text editors, including all standard hardware, uses an inverted-T arrangement for arrowkeys.

So, no problem, vim lets you remap your keys, which takes care of the problem. Many of us remap the "arrowkeys on the home row" to the standard inverted-T of ijkl (and remap i, for insert left, to h, reach left instead of reaching up), so that both sets of arrowkeys on the keyboard match each other and match their arrangement in every other app we use. Those with other keyboards, such as Dvorak, will often map to different keys that make more sense on their own keyboards. The ability to do so has always been one of the features of "real vi(m)".

I hope that for your file browser, you'll pass the keypresses through the (g)vimrc map, so that whatever keys a user has chosen for up, down, left, right, escape, and whatever else the file browser handles, will work the same way in both the editor and the file browser, which is to say, the way their "real vim" works.