People bought Android instead of WebOS and killed the most open platform mobile has ever seen. Hell, you could boot a kernel over USB, and Palm gave you the tools and commands to do it. On their official website.
Google does the same with its Nexus devices[1] in that you can build a completely open-source version of Android except for certain hardware-specific binary blobs.
The hard driver parts are for things like LTE, GPS, GSM, wifi, camera, etc. which the device manufacturers haven't chosen to open-source. I don't see how it's relevant to Android itself, any other OS would have the same problems with that hardware.