I feel like it's worth it to remember people's names. I also don't think there's any evidence that indicates being better at remembering one thing makes you worse at remembering others. In fact it might even make you better, as you're sharpening your brain, so to speak. (If there's evidence to the contrary please feel free to share.)
There is some evidence that great memory comes at the cost of other functions like creativity. However, that may simply be because people with poor memory spend more time grasping at straws.
