Any of the puzzle pieces individually aren't bad, it's putting them all together then adding additional pipes to stuff like (as you mention) SPAM filtering, etc. that make it such a huge pain in the ass. In an ideal world, there would be some package available where you simply specify the credentials for the machines you want to use for each piece (smtp, pop3, imap, secure pop3/imap interface, etc.) and it will setup everything for you.

In an ideal world.