RAG definitely is helpful! Fine-tuning imo is extremely powerful but it's still relatively alchemy - technically gpt4, Claude any large model is a finetune of a base model! Reasoning finetuning is also very powerful!

Tbh the hardest part is the lifecycle - ie new data, updating, serving etc - that seems to be the biggest issue