It's a bit weird that those niche books are sold - as opposed to just published free with maybe print-on-demand for those who want hard copies.
The authors don't make (practically) any money out of those books. Even the publisher probably doesn't. Vast majority of books aren't written to get any significant money (from sales at least), unless the author is delusional.
It's probably a kind of status thing and cultural lag. To get a publishing deal is a "stamp of approval" that it's a "real book" from a "real author" that has passed the gatekeeper. And that a copy of it costs something means it's not worthless, unlike e.g. the Linux kernel.
It's a combination of status and an attempt at an investment, and I guess somewhat of a lottery ticket.
There are also costs involved in a quality release even of a self-published novel. You want a cover. It's not expensive, but unless you're a talented artist it will show if you do it yourself. You can avoid paying for an editor, but it will generally be noticeable (you might have near perfect grammar and orthography, but you will miss things). If a publisher is going to put their name on it, those things will be at least a bit more expensive, and people tend to want to at least try to recoup some or all of costs even if they don't expect a significant return.
And if you want to turn writing into something more, you want to build an audience, and you want at least some signal of how large a portion of that audience are actually people prepared to pay, so that maybe next time your publisher is willing to put a little bit of money into marketing.
But yes, a lot of books could just as well be free. But vast quantities of writing - including fiction - is also released for free. Turns out there's signalling the other way too. It's hard to get people to read anything, but for at least some parts of the market, it's harder to convince them something free is worth their time.
The authors don't make (practically) any money out of those books. Even the publisher probably doesn't. Vast majority of books aren't written to get any significant money (from sales at least), unless the author is delusional.
It's probably a kind of status thing and cultural lag. To get a publishing deal is a "stamp of approval" that it's a "real book" from a "real author" that has passed the gatekeeper. And that a copy of it costs something means it's not worthless, unlike e.g. the Linux kernel.