> I don't think we should really take so much inspiration from the brain. We didn't make airplanes work by building bird machines so why should we do that here.
It’s not that we should mimic the brain’s implementation, but we should certainly strive to match the brain’s capabilities. One of its outwardly observable capabilities is that it is extremely efficient in the size of the training data set it requires.
Efficiency isn’t an implementation detail, it’s definitional to what “highly intelligent” means.
GPT-3 is not an airplane, it’s a zeppelin. Zeppelins also have scaling laws dictating that a zeppelin should be very very large. Building bigger and bigger zeppelins is one thing, justifying expending resources on gigantic zeppelins by stating the scaling law and concluding that a jet aircraft will magically pop out if you build a big enough zeppelin is quite another.
Your earlier analogy kind of feels like saying that because you can go further by adding more fuel to a jet engines fuel tank that you have failed at efficiency and should redesign the engine.
But generally I think the better analogy is a rocket ship. If we can still go higher and faster with more fuel we should try to do that before we worry about engine efficiency. You have to get to the moon before you can colonize the galaxy.
It’s not that we should mimic the brain’s implementation, but we should certainly strive to match the brain’s capabilities. One of its outwardly observable capabilities is that it is extremely efficient in the size of the training data set it requires.
Efficiency isn’t an implementation detail, it’s definitional to what “highly intelligent” means.
GPT-3 is not an airplane, it’s a zeppelin. Zeppelins also have scaling laws dictating that a zeppelin should be very very large. Building bigger and bigger zeppelins is one thing, justifying expending resources on gigantic zeppelins by stating the scaling law and concluding that a jet aircraft will magically pop out if you build a big enough zeppelin is quite another.