Japanese is hard for different reasons but some of them are self-inflicted. Bothering with pitch accent is one: no second of effort towards it will be ever relevant in comparison to learning new vocabulary or collocation. An other big issue is indeed in manuals, which teach sentences construction with the English logic instead of the Japanese one (eg starting with 私は学生です already poses 3-4 pedagogical problems). I’m working on a conversation guide (in French for now) to avoid those issue but that’s a lot of work...
Disagree re pitch accent being unimportant - it depends what your goals are and what your experience level is. If you’re competent enough with grammar but screw accent up constantly Japanese people get tired of listening to you, when you nail it people are more at ease talking to you in my experience. Like someone messing up stress in English all the time.
If your goal is to get over that then it’s absolutely worth studying, and arguably it’s worth studying early before you pick up bad habits in the first place.
As the parent mentioned, it’s all dependent on your goals.
