I don’t think people always do deductive reasoning when they attempt to do it. In fact I think people largely do abductive reasoning, even when they attempt deductive reasoning. Machines are better at deductive reasoning because sans some special purpose approach they can do nothing but follow the rules.
This is specifically why I think LLMs are so enchanting to humans, because it’s behavior and logic is more less sterile and more human in nature precisely because it’s a “most likely” based on its training data approach. With lots of examples of deductive reasoning it can structure a response that is deductively reasoned - until it doesn’t. The fact it can fail in the process of deductive reasoning shows it’s not actually deductively reasoning. This doesn’t mean it can’t produce results that are deductive - it’s literally unable to formulate a sense of rules and application of those rules in sequence to arrive at a conclusion based on the premise. It formulates a series of most likely tokens based on its training and context, so while it may quite often arrive at a conclusion that is deductive it never actually deduced anything.
I feel like you feel I’m somehow denigrating the output of the models. I’m not. I’m in fact saying we already have amazing deductive solvers and other reasoning systems that can do impressive proofs far beyond the capability of any human or LLM. But we have never built something that can abductively reason over an abstract semantic space, and that is AMAZING. Making LLMs perform rigorous deductive reasoning IMO is a non goal. Making a system of models and techniques that leverages best of breed and firmly plants the LLM in the space of abstract semantic abductive reasoning as the glue that unites everything is what we should be focused on. Then instead of spending 10 years making an LLM that can beat a high school chess champion, we can spend two months integrating world class chess AI into a system that can delegate to the AI chess solver when it plays chess.
This is specifically why I think LLMs are so enchanting to humans, because it’s behavior and logic is more less sterile and more human in nature precisely because it’s a “most likely” based on its training data approach. With lots of examples of deductive reasoning it can structure a response that is deductively reasoned - until it doesn’t. The fact it can fail in the process of deductive reasoning shows it’s not actually deductively reasoning. This doesn’t mean it can’t produce results that are deductive - it’s literally unable to formulate a sense of rules and application of those rules in sequence to arrive at a conclusion based on the premise. It formulates a series of most likely tokens based on its training and context, so while it may quite often arrive at a conclusion that is deductive it never actually deduced anything.
I feel like you feel I’m somehow denigrating the output of the models. I’m not. I’m in fact saying we already have amazing deductive solvers and other reasoning systems that can do impressive proofs far beyond the capability of any human or LLM. But we have never built something that can abductively reason over an abstract semantic space, and that is AMAZING. Making LLMs perform rigorous deductive reasoning IMO is a non goal. Making a system of models and techniques that leverages best of breed and firmly plants the LLM in the space of abstract semantic abductive reasoning as the glue that unites everything is what we should be focused on. Then instead of spending 10 years making an LLM that can beat a high school chess champion, we can spend two months integrating world class chess AI into a system that can delegate to the AI chess solver when it plays chess.