1. We've learned that relu is a really good function to use for neural networks.
2. We've learned that regularization is essential to having large networks without overfitting.
3. We've learned how to scale models up and down so that you can train something on a giant cluster then run it on your phone.
4. We've learned how to apply this to lots of specific kinds of problems.
etc, etc, etc.
We do NOT simply do what we did in the 80s then throw more hardware at it. Instead we are getting better at making the hardware do something useful.
1. We've learned that relu is a really good function to use for neural networks.
2. We've learned that regularization is essential to having large networks without overfitting.
3. We've learned how to scale models up and down so that you can train something on a giant cluster then run it on your phone.
4. We've learned how to apply this to lots of specific kinds of problems.
etc, etc, etc.
We do NOT simply do what we did in the 80s then throw more hardware at it. Instead we are getting better at making the hardware do something useful.