I'm reading the Cambridge Grammar of the English Language at the moment. It has a complex layout with varied typography, examples with subscripts and superscripts, text that's left, center, and right aligned, callout boxes, sidebars, diagrams, tables, multiple heading levels, footnotes and so on. ePub simply can't handle complex layouts or typographic requirements beyond those you'd find in a novel. Also, the justification and hyphenation algorithms in ePub readers are typically complete crap.
Yes, I find there are two kinds of books, those that would work as audio books and those that wouldn't. Those that work as audio books also work well as ePub and, in fact, are better in ePub form than as PDF, because I have more control over the presentation.
Books that wouldn't work very well as audio books usually won't work well as ePub either. Charts, maps, complex typography, etc., that would be lost in audio form is mostly lost by ePub. I don't know how much of that loss is due to to ePub format, how much due to ePub viewer apps, how much due to ePub creation tools, and how much due to creators just not trying very hard, but the combination makes most high-quality textbooks much better as PDF, which "beach literature" is better as ePub.
I think ePub's can handle everything you mentioned, ePub's are just XHTML, and a subset of css [1]. Not sure about callout boxes, I don't know exactly what subset of elements are available as I've never written one, but everything else you've mentioned looks to be available [2].
Fun epub file trick: rename the file .zip, unzip -a yourbook.zip (double click unzip doesn't work on osx for me for some reason) and check out the html, css, images and xml of your book.
While ePubs can handle more complicated layouts, it's much easier to take an existing print book layout and export to PDF and have it retain the formatting than it is to do the same with ePub. So to export the same layout to ePub, you basically need a web designer to go over it and mark up the parts correctly and write the right CSS. Exporting to PDF is pretty much one step and you're done.
>the justification and hyphenation algorithms in ePub readers are typically complete crap.
IMO, ePub readers have the best justification algorithm: flush left, and the best hyphenation algorithm: never hyphenate. PDFs often force full justification on you, which makes it harder to read because all the line endings have the same spacing, making it easy to lose your position in the text.
I actually doubt that "flush left only" and "no hyphenation" are the only, or even standard, options for most readers.
But I think there's a synthesis possible between what you mean and what OP meant: ePub readers are somewhat better at adjusting to the reader. "Reader" here in terms of both the device and the actual human. Text can reflow, columns are used only where they make sense, and settings can be changed.
PDFs are better at allowing the creator to, well, create. I know there's a hard core demographic that doesn't believe in any sort visual design. Yet there are topics and authors where choices of placement, font, color etc. are made with intent, and to good effect. Everything Tufte comes to mind.
It should be possible, with something like Apple's Book Author, to create such designs that work well within the e-Book format. But I haven't seen any examples, probably because I rarely read textbooks these days and tend to buy paper copies of such books, anyway.
In as far as I have seen illustrations in non-PDF formats, the results are dismal, with all sorts of sizing problems etc.
> making it easy to lose your position in the text
I don't have that problem. I find find ragged-right with no hyphenation less pleasant to read and to look at. When I'm reading a novel on my Kindle, I have it set ragged-right because the Kindle doesn't justify well (just word spacing I think). Same on the web. But when I can get good justification and hyphenation, like on a properly typeset PDF or physical book, I prefer it.