Agreed. My experience from watching some...challenging films in small film festivals is that bad picture can be tolerated, while bad audio will utterly ruin a film.
Just about every resource (books, websites, videos...) for indie film makers makes a point of stressing the absolute importance of good audio, but it's still often ignored.
As you say, people will forgive bad image quality before they will forgive bad audio.