I have this deep seated fear that NNs will be the death of the lessons learned from 1970-2010. After all, if you can use massive amounts of compute to materialize what seems to be a good enough function approximator, why do advanced algorithms at all?
Obviously the reason we should is that approximators like the NNs have explainability issues and corner case unpredictability issues plus they are bad at real world complexity (which is why self driving efforts continue to struggle even when exposed to a narrow subset of the real world).
I think you're right on about explainability and unexpected handling of corner cases - but I think one of the lessons from GOFAI is that handcrafted algorithms might look good in a lab, but rarely handle real-world complexity well at all. Folks worked for decades to try to make systems that did even a tiny fraction of what chatgpt or SD do and basically all failed.
For safety stuff, justice-related decision-making, etc I think explainability is critical, but on the other hand for something like "match doodle to controlled vocabulary of shapes" (and tons of other very-simple-for-humans-but-annoyingly-hard-for-computers problems), why not just use the tiny model?
Maybe if we get really good at making ML models we can make models that invent comprehensible algorithms that solve complex problems and can be tweaked by hand. Maybe if we discover that a problem can be reasonably well solved by a very tiny model, that's a good indication that there is in fact a decent algorithm for solving that problem (and it's worth trying to find the human-comprehensible algorithm).
I have this deep seated fear that NNs will be the death of the lessons learned from 1970-2010. After all, if you can use massive amounts of compute to materialize what seems to be a good enough function approximator, why do advanced algorithms at all?
Obviously the reason we should is that approximators like the NNs have explainability issues and corner case unpredictability issues plus they are bad at real world complexity (which is why self driving efforts continue to struggle even when exposed to a narrow subset of the real world).