Are you selling your ability to recite stuff? Then certainly.

there are plenty of open source LLMs trained on harry potter, is that fine?

No

I pay for a service. The service recites a novel to me. The service would need permission to do this or it is copyright infringement.

This is an extremely common strawman argument. We're not discussing human memory.