I'd try jellyfin first to see if it fits your needs.
I went with plex years ago, because they had good app support on the various devices in my house. (Mostly roku now)
The problem with Plex is:
1. during a recent half day internet outage (during prime-time) I was unable to use plex because the app didn't have access to the internet. The network was all up and running, devices could see each other, but plex decided that even though the media was on the local network it wasn't good enough and refused to finish playing the video we were watching. (The internet went out 20 minutes in)
2. Plex the company has gone fully into adding all kinds of streaming services in order to make a buck. While you can remove these things from your menu, it is just annoying.
3. Plex doesn't always fix known issues. Over the years I've run across several issues in plex that after trying to troubleshoot find that it is a known issue Plex refuses to address. For example, I've recently had some issues with some videos dropping half or more of the frames while the audio is fine. Turns out, plex doesn't like something in the files metadata and this is the result. Plex is the only one that has the issue with the file. It plays fine locally with VLC and streams fine with other programs.
I was reluctant to switch from Emby to Plex for this same reason but it turns out you can run plex self-hosted without any account. I have Plex running on a server and streaming from the Plex app on an LG TV without requiring an account.
I'll warn you that Plex will do everything possible to get you to add an account. One update (several years back) locked me out of making any changes to my server until I created an account.
I've not used Plex, so I'm not sure. You'd need to find a way to expose it to the Internet (mine's only on my local network) but that shouldn't be too hard. Just forward the correct ports on your router.
It does have an account system, including the ability to restrict which "libraries" an account can access, which is great if you have kids. For adults, it lets you track your viewing progress/status separately, just like having multiple Netflix profiles.
One thing to account for is that it has to transcode and/or remux videos for clients that can't handle a file's native codecs, audio or video, which can put a pretty heavy load on the server. A Raspberry Pi or weaker x86 machine won't be able to do this without frequent pauses and frame-dropping, for any but very low-resolution media. Solutions to this include: 1) ensuring that your clients can all handle a huge range of codecs, so it never has to transcode (IME audio is, these days, trickier than video, especially ensuring things like Dolby Atmos are supported), 2) getting a really powerful server, in particular with a video card that Jellyfin can use for transcoding, and 3) falling back on just downloading the file and throwing it in VLC (the web interface makes it really easy to download the raw video files in a pinch, though if you have big high-quality 4K rips they'll come down at full size, which can be inconvenient on devices with limited storage, like, say, iPads).
However, I think Plex or anything else will have similar limitations, since they all have to do something like that to accommodate players & devices with limited codec support.
Jellyfin's been very stable for me, which is part of why I'm still on it. I also find the UI in most of their clients much, much more to my liking than something like Kodi. But IDK about Plex.
[EDIT] Oh, I guess you could also batch-job transcode all the files to something very widely-supported, outside of JellyFin, though likely at some cost in quality and maybe also file size. Plus it'd probably take at least an hour or two to hack together a script to do it, for a wide range of input codecs.
I have primarily used Plex and pretty much everything you said is accurate for Plex as well. Limited transcoding based on the machine it is running on. As disc has become cheaper, I have pretty much stopped doing batch transcodes, which is great for the most part. But there are definitely negatives when you want to watch something offline, or remotely. Biggest pain point is subtitles though. Since they aren't ripped as text and then sent to a client, they have to be burned in to the video itself and transcoded on the fly. Which means losing out on 'forced' ones if it can't transcode fast enough.
Plex has definitely started to try and commercialize itself more and offer other stuff, when all I want is access to my own media. So I may look into Jellyfin more soon.
As for batch transcode jobs, I had a system that I was able to set up as essentially a black box. Drop a rip into a folder and out the other side comes a smaller one at a reasonable quality. With forced subs burned right into the actual video. Mostly based on https://github.com/donmelton/video_transcoding
I currently use and host both, plex while non free is more friendly to less advanced users and has native iOS and apple tv apps (which jellyfin does not (the jellyfin ios app is a webview and dosen't always behave well for me))
There are 3rd party apps for jellyfin on apple tv, but it's just not as smooth of an experience as plex.
I'm hoping that jellyfin will push plex to get better, as some of the most requested features for plex have gone unanswered for years, which is quite frustrating for software that is paid.
Have a look at Infuse, it works with Plex, Emby/Jellyfin, and possibly SMB. It’s one of the best and high quality apps I’ve encountered on the Apple TV, and one of the best video players period. For me it completely eliminates the need for transcoding, it plays everything.
There’s also a bare bones native Jellyfin app for tv/iOS called SwiftFin, but it’s currently only (publicly) available in TestFlight.
