They also have an enormous downside compared to the other 'bad drugs' in that they leave you essentially non-functional for 3-24 hours depending on the drug. Don't get me wrong, I'm a fan of hallucinogens, but essentially all recreational drugs have a downside.
If you mean drugs like LSD leave you unable to work for 8 hours, this is probably true - although it depends what sort of work you are in (heavy machinery is out, painting is probably in). However, the same thing can be said for a day spent in a ski resort with your cellphone left behind.
The term "non-functional" has a pejorative connotation and implies humans have some teleological purpose beyond personal goals and desires. Recreation is subjective; it's not appropriate to call this a 'downside'.
I think it is appropriate to call it such. In all the times I've taken mushrooms I would have been completely unable to respond to an emergency situation. The same with LSD. No other drug I've taken would have left me that incapable, with the exception of extreme amounts of alcohol.
I'm not saying it's not fun or worthwhile, just that hallucinogens leave you impaired in a way other drugs don't.
I would tolerate a statement like "unable to respond to an emergency situation" - although, this varies substantially with the substance, quantity, person, and nature of the emergency. And the recognition that there are plenty of non-pharmacological situations that render you equally unfit for dealing with certain kinds of emergencies - good luck dealing with a back injury while skiing off-piste.
However, throwing out a statement like "drugs make you nonfunctional" does not help the discourse.