Ironically the tree outside my house was recently removed because the fruit annoyed people. Personally I thought the tree was well worth the hour or two a year it took to remove the fruit in autumn.