In my experience, Chinese people are thrilled when non-Chinese people speak their language (even poorly), probably because it's pretty rare.

But yes, the tones are the hard part. The grammar is straight forward and similar to English, and characters are (almost always) pronounced only one way.