> We usually assume that (a) the entire universe is computable and (b) even stronger than that, the entire universe is _learnable_, so we can just approximate everything using almost any function as long as we use neural networks and backpropagation, and have enough data.
I don't think the assumption is that strong. The assumption is rather that human learning is computable and therefore a machine equivalent of it should be too.
> The assumption is rather that human learning is computable
I don't think the assumption is even that strong! The skills that really set us humans above mere machines - i.e. causal inference, creativity, critical analysis, self-awareness - I don't think are assumed to be computable (IMHO there's no evidence as yet to suggest otherwise). The only skill that AI really currently possesses, is the ability to apply an ever-more-elaborate statistical aggregate function to data. The assumption is just that anything that can be encoded as data, can be operated on to produce an ever-more-elaborate aggregate result.
I don't think the assumption is that strong. The assumption is rather that human learning is computable and therefore a machine equivalent of it should be too.