In my own experience, every engineer that I have worked with who regularly uses psychedelics has substandard problem-solving abilities.
Note that I'm not "knocking" psychedelics per se. I've never personally tried them and I don't know enough to comment on their effects or risks. I'm just providing an anecdotal counterpoint to yours.
There are a lot of general comments in this thread about people's minds being usefully expanded by drug use. I'm not disputing these, but could someone perhaps provide a specific example about a problem they were stuck on which they then managed to solve with the assistance of drugs?
No way man, that's such a square question, it's not about specifics, it's about generalities.
Ahem. That was a joke. But in seriousness, I've also never heard a specific example of a programming problem solved during a trip, but I can say that I've found tripping to be very beneficial in some unexpected areas:
1. Holistic understanding feels easier. E.g. understanding other people on a much deeper holistic level. I'm usually not very great at group socializing in parties or whatnot but tripping actually helps with that a lot. This is, in my mind, the same kind of "systems understanding" that could be applied to programming. You can intuitively understand a complex system of subtle interactions, on a deeper level than just what's being said.
2. Visualization. Almost anything can be visualized, even if it doesn't really make sense. And the visualizations feel real, which makes them easier to move and comprehend. Sometimes you can literally see them, like in the patterns of the veins on your hand, or the stitching in your clothes.
3. Unexpected mutation. During a trip, things can often move in a weird way, including things that don't normally move, like concepts or diagrams (or walls -- just try not to stare at them too long). It's notoriously challenging to rotate, skew, or otherwise transform your mental visuals. Tripping can make this trivial.
4. Perceived reality. The difference between real and virtual completely fades. Video games feel real (which is awesome when it's not terrifying). Have you ever played an FPS where you are literally afraid to die? It actually makes you a lot better! This same perceptive shift collapses the difference between real/virtual, digital/analog (in the nontechnical sense), practical/theoretical. It collapses inside/outside as well. You are not separate from what you are doing or thinking about; you're part of the system, like you're literally contained within the conceptual structure. This kind of dichotomy-destroying mindset opens up some very interesting new perspectives.
Compilers are generally considered very complex programs, and the "mind expanding" properties of psychedelics helped him for laying out the high level design (he did not do the actual coding while under the influence).
I regularly consume MDMA with my best friend (two or three or four times a year) and during these day-long trips, we discuss a wide range of problems we're having with our personal programming projects. Not every problem is solved in these times, but there certainly is some ground gained on a few of them. Sometimes it might as well be NP-hard to solve concrete problems after enough MDMA, but I am confident that the effects of MDMA establish a channel of communication between us that we don't normally get to enjoy, and that channel can be very useful for solving specific problems. And the things we talk about range very widely to a huge number of subjects. I'd say most of our core beliefs about the universe are affected by our use of psychedelics. Personally, I also use mushrooms for sort of directed trips to explore my life situation, and again, it's pretty hard to do any coding while high, but conceptually, there is absolutely no way I would have had some of the ideas I've had without psychedelics.
To play devil's advocate: could it be that generally curious, risk-taking, meta-thinking individuals tend to make better engineers, which just happens to be a personality type that is naturally drawn to psychedelics?
Really, we can play arm-chair neuroscientist indefinitely, but there is only one way to get to the bottom of this. We must gather more, um, data.
To create innovative solutions you have to be creative. This includes thinking outside of the box, putting dogma into question, clearing your own paths, and this all translates to connecting concepts that we don't usually connect. Some people do it naturally, some use psychedelics, some do both.
I think cultivating this as an attitude is usually a good thing (in life) and can lead to powerful insights. In fact this is a really cheap way to innovate. Take concepts from one domain, cast them onto another, profit. A cross disciplinary culture helps a lot. I'm sure many people here understand very well what I mean.
So to answer you more precisely, I really have only one data-point and I think both the traits you describe and psychedelics can make interesting (if not better) engineers.
