Logic is an awesome tool that took us from Greek philosophers to the gates on our computers. The challenge with pure rationalism is checking the first principles that the thinking comes from. Logic can lead you astray if the principles are wrong, or you miss the complexity along the way.
On the missing first principles, look at Aristotle. One of the history's greatest logicians came to many false conclusions.
On missing complexity, note that Natural Selection came from empirical analysis rather than first principles thinking. (It could have come from the latter, but was too complex) [1]
This doesn't discount logic, it just highlights that answers should always come with provisional humility.
The ‘rationalist’ group being discussed here aren't Cartesian rationalists, who dismissed empiricism; rather, they're Bayesian empiricists. Bayesian probability turns out to be precisely the unique extension of Boolean logic to continuous real probability that Aristotle (nominally an empiricist!) was lacking. (I think they call themselves “rationalists” because of the ideal of a “rational Bayesian agent” in economics.)
However, they have a slogan, “One does not simply reason over the joint conditional probability distribution of the universe.” Which is to say, AIXI is uncomputable, and even AIXI can only reason over computable probability distributions!
Bayesian inference is very, very often used in the types of philosophical/speculative discussions that Rationalists like instead of actual empirical study. It's a very convinient framework for speculating wildly while still maintaining a level of in-principle rationality, since, of course, you [claim that] you will update your priors if someone happens to actually study the phenomenon in question.
The reality is that reasoning breaks down almost immediately if probabilities are not almost perfectly known (to the level that we know them in, say, quantum mechanics, or poker). So applying Bayesian reasoning to something like the number of intelligent species in the galaxy ("Drake's equation"), or the relative intelligence of AI ("the Singularity") or any such subject allows you to draw any conclusion you actually wanted to draw all along, and then find premises you like to reach there.
They can call themselves empiricists all they like, it only takes a few exposures to their number to come away with a firm conviction (or, let's say, updated prior?) that they are not.
First-principles reasoning and the selection of convenient priors are consistently preferenced over the slow, grinding work of iterative empiricism and the humility to commit to observation before making overly broad theoretical claims.
The former let you seem right about something right now. The latter more often than not lead you to discover you are wrong (in interesting ways) much later on.
Who are all the rationalists you guys are reading?
I read the NYT and rat blogs all the time. And the NYT is not the one that's far more likely to deeply engage with the research and studies on the topic.
Logic is the study of what is true, and also what is provable.
In the most ideal circumstances, these are the same. Logic has been decomposed into model theory (the study of what is true) and proof theory (the study of what is provable). So much of modern day rationalism is unmoored proof theory. Many of them would do well to read Kant's "The Critique of Pure Reason."
Unfortunately, in the very complex systems we often deal with, what is true may not be provable and many things which are provable may not be true. This is why it's equally as important to hone your skills of discernment, and practice reckoning as well as reasoning. I think of it as hearing "a ring of truth," but this is obviously unfalsifiable and I must remain skeptical against myself when I believe I hear this. It should be a guide toward deeper investigation, not the final destination.
Many people are led astray by thinking. It is seductive. It should be more commonly said that thinking is but a conscious stumbling block on the way to unconscious perfection.
I'm just going to defend Aristotle a bit. His incomplete logic and metaphysics nevertheless provided a powerful foundation to inquire into many aspects of the world that his predecessors did not do, nor do systematically. His community did not shy away from empirical research in biology. They all came to wrong conclusions in some things, but we should rather fault their successors for not challenging them.
You stated one of their core doctrines, something they’ve been loudly preaching for as long as they’ve existed, as though it was something they disagree with and had never even considered. Can you blame them for a bit of exasperation? For wanting to simply downvote-and-disengage from someone who makes up falsehoods about them and then gloats about how it annoyed them? Life is too short to tilt at windmills.
On the missing first principles, look at Aristotle. One of the history's greatest logicians came to many false conclusions.
On missing complexity, note that Natural Selection came from empirical analysis rather than first principles thinking. (It could have come from the latter, but was too complex) [1]
This doesn't discount logic, it just highlights that answers should always come with provisional humility.
And I'm still a superfan of Scott Aaronson.
[0] https://www.wired.com/story/aristotle-was-wrong-very-wrong-b...
[1] https://www.jstor.org/stable/2400494