I have been doing open source for 4-5 months now (main project has 1.8k stars on Github so far), so I am a newbie. However, I have always found ignoring people to be difficult. I immediately get a sense of guilt when doing it.
Likewise, although a few more stars and a little longer doing it. If you're interested in peering into the future, you could look at my story over at GitHub's README project for maintainers - https://github.com/readme/alex-ellis
Keep it up and good luck. The most you invest in community, the more you can delegate and find "co-developers" who can help you in ways you couldn't imagine.