Author has it completely backward. [edit: on further reading, the title is clickbait but the article content is consistent with my point below)
With only a few exceptions, ML is incredibly simple (there is no AI). The math is simple, the mechanics of evaluating it is simple, the reason it works is simple, and it only really works well if you have absurd amounts of data and CPU time.
Physics is... determining the mathematics you need to know on the fly while discovering and explaining many phenomena.
You can spend decades focusing on matrix multiplications and other fairly straightforward trivia to analyze your particle trajectories, and then suddenly, you need to know group theory or some completely different field of math just to understand the basic modelling.
Personally I think the most impressive thing in physics and stats so far is our ability to predict the trajectories of solar system objects far into the future. After some very serious numerical analysis over the past 50 years, we've reached the point where there aren't many improvements we can make, and most of them come from identifying new objects, their position, and mass (data/parameters), and the real argument is about whether the underlying behavior is truly unpredictable even if you have perfect information.
in fact many of my friends who were physics people, when I show them the code of a large scale batch training system they wonder why they did physics instead of CS because the math is so unbelivably simple compared ot the tensor path integrals they had to learn in Senior Physics.
I didn't miss the point. my graduate work was using constraint optimizers in molecular dynamics (n-body physics) and it translated to ML (I didn't have to relearn anything). The one part that was truly simpler is the objective functions in ML are believable to be convex, while n-body physics with constraints are highly non-convex).
The funny part is that Group theory is heavily integrated with the mathematics of neural networks. So much so that some mathematicians say that finding the invariants of groups is what makes NNs work.
The title is 'clickbait' for sure, but not neccessarily incorrect. After all 'simple' can mean different things to different people. And the author clarifies what he means by 'simple' in the rest of the article.
Physics isn't simple. It took (literally) thousands of years of study by very smart people to get to the point where what we call "intuition" about the physical world is what it is. And, as any physicist who paid attention in class will tell you, even that intuition isn't really right.
Any simplicity observed in physics is born of long familiarity, or is the result of underlying complexities being masked or approximated away.
Physics is simple in a certain sense and the paper explains this. In theory the force acting on something you drop could be a function of the state of every particle in the universe and each one could contribute with a different weight. But this is not the case, things are only influenced by things nearby and the weights involved are not arbitrary, the charge of each electron, for example, is the same. In that sense physics is unbelievable simple as compared to what it could be.
We also don't know for sure the "constants" are constant throughout the universe (spatially and temporally) This is assumed for now, and seems almost certainly true.
I think it's unsafe to assume the above are Absolutely True and that that's why physics is simple.
nobody has proved the conjecture "things are only influenced by things nearby" this is one of the largest ongoing arguments in QM
This is a much more nuanced matter. Non-locality as in global wave function collapse or Bohmian mechanics has not the same consequences as classical non-locality, there is no causal influence from things you do to an entangled particle at the other end of the universe. Also to entangle particles they have to first interact locally before they can be separated.
We also don't know for sure the "constants" are constant throughout the universe (spatially and temporally) This is assumed for now, and seems almost certainly true.
This does not really change the argument, even if, for example, the fine-structure constant is not constant after all, then there will most likely be a hand full of other constants that describe how it varies over space and time. This is very different from every electron having a unique electric charge that is not governed by anything and the only way to figure it out is to measure it for each electron.
It would also probably make a difference how values are distributed, are the charges of the electrons nicely distributed and vary by a factor of two or ten? Some statistical theory could probably deal with that. But what if there is no nice distribution, if values vary by hundreds of orders of magnitude, if the expectation value or the variance of the charge is infinite? I am certainly unqualified to make any definitive statements but I can imagine physics to be weird to the point that it becomes mathematically intractable or at least only produces useless answers because of the amount of uncertainty that enters the equations.
In any case, we know that the universe is simple to a very good approximation, even if fundamentally everything depends on everything and the electron charges are all random, those effects are small and we can have a good approximation with simple theories.
"Please note that the notion of simplicity to which we are appealing is not at all meant to suggest that physics is trivial. Instead, we mean it as a compliment: we have the utmost respect for the work of physicists. This essay is an apologia for physicists doing machine learning qua physicists; it is meant to interrogate what it is about the approach or perspective of physicists that allows them to reach so far in explaining fundamental phenomena and then consider whether that same approach could be applied more broadly to understanding intelligence"
Physics is not simple. e=mc^2 might look nice on a t-shirt but it is a cherry picked formula. Newtonian mechanics you learn in high school might seem simple to someone with a MINT degree but they describe a small part of the world, and complicated things like friction are discarded.
General relativity, quantum chromodynamics, etc. They are all incredibly complicated.
Even bog standard classical mechanics can be very complicated and unintuitive, once you go to things like Hamiltonian dynamics and many-body problems. Then you have statistical Physics and, as you said, relativity and quantum mechanics. I’ve recently spent some time learning about some quantum gravity theories; these things are hard.
Yes. Chaotic systems arise in classical mechanics. So while often the formalism is easier to write down than for example in general relativity, calculating a solution is often very difficult or practically impossible.
It's probably worth distinguishing between conceptually simple and mechanically simple. A many body problem is easy to understand (relatively), you're just extending existing rules onto more things at once. Now, actually calculating a position at a time given some initial vectors? That's complicated.
You’re right. But many-body can also turn into statistical physics, which is quite complicated conceptually (basically when entropy and ergodicity become important). Then you have the best of analytical Mechanics, statistics, and quantum mechanics if you fancy.
It was a while back but I think we derived it in school aged 16-17 in first year A level Physics (UK.) Is that roughly "junior year in college" elsewhere?
As I recall, deriving it required a solid grasp of special relativity and calculus. I mean, if you want to say that you did that at 16, I won't argue with you. But if you claim all students did that it seems unlikely.
Edit: Oxford University, since you are from the UK and that's reputable within the country, lists General Relativity (and hence E=MC^2's derivation) as a 3rd year course in the Physics program. So that makes it squarely in the junior year.
physics is also mostly deterministic (attach probability distributions for stochastic/quantum stuff) and there are well defined rules (energy, symmetry, noether, etc).
at the end of the day ai has some space for models and so does physics. because physics has well defined rules it's easier to apply constraints to that space vs ai/ml where it's informed guesswork.
of course there will be a correspondence between parameters in a model and emergent physical phenomena ... and i'm sure really nice scaling laws, etc will come out , this is just coarse graining.
onsager and the likes were onto this stuff way before deep learning was a thing. i think this connection is uninteresting because optimization in it's heart is physics. dl is just one aspect.
- a physicist (escaped to the greener pastures of swe, shame really, i miss it but not the wl balance)
Most of the responses here seem to imply that the author doesn't understand that physics can be complicated (in the sense of being hard to learn or having big equations). He studies theoretical physics at MIT [1], so I expect he does.
On the content: it's pretty weird that our best models don't use much of the world's underlying structure at all. State-of-the-art vision models like vision transformers and MLP-mixer do just fine when you shuffle the pixels. You could argue that modern image datasets are so big that any relevant structure could be learned by attention, but it still feels like we're doing _something_ wrong when pixel order doesn't matter at all ¯\_(ツ)_/¯
Vision models need the pixel ordering to match the one they have been trained on, in order to work.
They won't generalize after training to transformations of the data that they haven't been trained on, even simple ones such as rotations, whereas humans will.
So I would argue that vision model do use the "underlying structure", and even that one of their problems is that they make use of some of the "underlying structures" that are not actually important, such as image luminosity, rotations etc. I think people usually augments the data with these transformations beforehand during preprocessing to enforce invariance.
Physics is explaining what is, in a fairly testable domain.
AI is figuring out how to replicate something fuzzily understood, from a difficult to test domain, using techniques that are largely in a different domain altogether, because even for the parts of the inspirational domain we kinda-sorta understand, we don’t have the tools to directly reproduce them easily so at best we simulate them in alternative media.
(“Physics is simple” overstates the case, still, but “AI is hard” understates the case, so, relatively speaking, its something of a wash.)
Physics has had a 470 year head start, if you measure from Copernicus. Of course it looks simple. It wasn't simple at the time; it's taken something like 14 generations so far.
A fun read, but the title is somewhat misleading. Excerpts from the paper:
This is why (understanding how) AI (works) is hard.
This is why physics is (able to offer) simple (explanations for seemingly complicated natural phenomena)
The author is arguing that:
1. "top-down analysis" to deep learning is ineffective due to the nature AI.
2. It should be possible to use the "sparsity" principle from physics as "a guiding principle of deep learning theory"
Where the "sparsity" principle dicit:
The number of parameters defining a theory should be far less than the number of experiments that such a theory describes
..
Personally, I'm skeptical. I prefer to understand whole theater of ANN as an attempt to optimize neural-net for practicality. Even deep learning is a technique for structuring the network for efficient training.
I thinks so because even the most dumbest neural network - layers with high connectivity - can perform certain tasks, if we train it really really hard with fingers crossed. To avoid uncertainty and improve performance, it's necessary to structure the network. This cuts unnecessary computation, and prevents over-fitting and over-labeling.
That is, we have to inject bias to network, and this makes any bottom-up analysis biased and less generalizable.
As someone with a physics degree, physicists really have a unique ability to stroke their own egos. I think I agree with the premise - structure is certainly the most important thing in our physics and machine learning. However, a significant portion of the effectiveness behind ML is letting the computer find the structure for itself rather than doing it yourself. The most effective uses guide the learning via minimal structure.
The problem the article is basically describing is that many neural networks do not utilise all the information available, or are over trained, or are not in a sense compact enough. There is redundant information.
For example recognising pictures of cats, the fact you can rotate a cat picture and its still a cat isn't necessarily baked in. If you visualise layers there seems to be a lot of redundant information, (if you train on rotated cats you get layers that look like rotations of each other). If there is a mechanism to store this kind of information more compactly, I am not aware of it.
Perhaps there is the idea that some kind of structure that encodes the redundant information, should be emergent or natural, it should evolve out of the models without having to make arbitrary choices. On the other hand maybe we should turn to physics and add these physical kind of constraints, there is a way to renormalise or make use of symmetries to improve our current models. Perhaps a neural network is more like an algebraic variety (collection of polynomials) with a group action.
To make a stab at the question posed by the title:
We've known the principles of Newtonian physics since, well, Newton.
We'll need an Isaac-Newton-level of insight into intelligence before we can make it so simple.
Nevermind that we can't even seem to agree on the definition of intelligence.
As to the definition, I have an objection to calling the current big-data stats we do nowadays "machine learning" or "artificial intelligence". There is no intelligence there.
Rather, to allude to the article, I consider it more "machine muscle memory" or "artificial intuition". The algorithm can tell you, "I have a really good hunch about this based on the zillions of examples I've seen" but it can't derive underlying truths to reason about why.
Perhaps we are recapitulating phylogeny when it comes to artificial neural systems? We have something like an autonomous nervous system, and a brainstem, but we need so much more to get to intelligence.
The single most successful program of physics is quantum mechanics, and it is neither intuitive nor simple. Relativity, while conceptually simple, isn't so simple either and is far from intuitive (consider that momentum is conserved during a relativistic near-miss collision only if one considers the entire collision over a sufficiently length period, since at any moment the force vector between the bodies does NOT align with separation vector, since the force acting at one moment was exchanged, at c, when the bodies were in different positions).
There are a lot of simple concepts in physics, many of them basically teaching aids to get people started. When one gets deep into the field, simple and intuitive go by the wayside.
I read TFA, hence the reference to intuition. There is nothing in the article that makes a compelling case of physics being simple, other than rhetoric.
We forget at our peril the Michelson-Morley Experiment and the Ultraviolet Catastrophe, and if we forget these, we may assume now too that we have it all figured out.
Of course, active researchers in the subject, both theoretical and experimental, do not forget.
Titles have to be short, and as such they can't hope to represent the contents of the article completely accurately.
If you wanted to do that you would have to make the title equal to the article's contents.
Based on the parts which I've read so far, a more accurate title would be 'Why some currently hot parts of AI not well understood, and some parts of Physics well understood?'
I think the original title is an ok approximation of this.
The author is talking about how a given physics model appears simple when they are presented with it, e.g. a particular quantum field theory. This is the kind of limited perspective about research that an undergraduate physicist may develop simply by solving the hand crafted problems that are presented to them.
However, the true difficulty in physics is arriving at that model in the first place. Decades of work offered up against experiment, the associated conceptual leaps in understanding required to get to e.g. a quantum field theory which succesfully predicts things are nothing short of a monumental achievement. To say that physics is simple is ludicrous.
You are missing the point the article. Author is not trying to argue that AI is 'harder' than physics, like a freshman cs major might argue with their physics friends.
Author is talking about how our physical theories, such as QFT, currently have more predictive power than any theories we currently have about machine learning/deep learning.
While QFT makes some amazingly precise predictions in certain areas like the fine structure constant, it is nearly useless for predicting even most chemistry.
In practice, the computations required to use the QFT model are just too complex for modern computers when it comes to single atoms with more than a few protons, not to mention larger molecules. Instead, we must use simplified models like the Bohr model to make predictions about molecular bonds.
This actually seems to be very similar to AI where we understand not everything, but a lot about basic neurons, yet the emergent phenomena of intelligence is very difficult to predict due to the explosion of computational complexity.
I think the article misses the point of what physics is. It is not a collection of "sparse" models and principles, rather, it is a scientific discipline from which such models have emerged.
You will notice the article conflates the two things: physics and the known laws of physics (e.g. first para in section 1.2). Simplicity of the latter does not imply simplicity of the former, but the article assumes that it does in order to tackle/state the question as posed: "Why is AI hard and physics simple?".
IMO if all it takes is a few simple substitutions like “physics” -> “known laws of physics” to make the article or title make sense, then it’s unfair to say that the author has missed the point. C.f. Reading the strongest possible interpretation and all that.
for what it is worth: physics is not easy, never was, never will be. the painfully slow invention (or is it discovery?) of the mathematical machinery required to understand physical phenomena is one of the most astonishing feats of the human brain. In my view it sets it apart from anything remotely "AI"-sh.
Starting with early calculus, to differential geometry, Hilbert spaces and whatnot, the brain doesn't fit models to data, it is making up categories and concepts (new classes of metamodels if you wish) as it goes along (as improved experimental devices augment our sensory inputs). To cope and explain these totally indirect information flows the brain conjures up symmetries in invisible abstract spaces, invariants and visual imagery from alien internal worlds, pursues "thought experiments" to restore sanity... Machine learning style fitting of "model" to data is just one of the final steps. Important but hardly defining the process.
The "AI" folks oversold their goods by a breathtaking factor and are now exposed to the entire planet. No physicist will ever be able to bail you out :-)
> Ignoring the mathematician’s appeal to rigorous
(re)formulation, we focus on the latter part of the quote: “The physicist’s often crude
experience leads in an uncanny number of cases to an amazingly accurate description of a
large class of phenomena.”
Non-practitioner in either field ...
If you don't focus on the more or less "defined" ML subset, and focus on the AI in the question, whatever "AI" has meant over the years, then ...
The reason why AI is hard and physics simple may be that physics already "is," while we may not yet have defined what AI should be and what to do with it .
Physics has guardrails of reality, however subtle and hidden they may be. What is described by physics is what it is, and will be there as it is, whether we correctly describe it or not, and whether we observe it or not. (Depending on your view of an observed universe's dependant relationship with its observer.)
AI is ... what? In any particular decade? We aren't describing anything that already is, and that comes with constraints. We're making it up.
Specification is hard.
The universe as described by physics, on the other hand, is well beyond specification and requirements gathering. We're just writing the missing manual.
I call on theoretical physicists to do theoretical physics. “AI” has already attracted huge number of people (driven by meaningless buzzwords, and hoping to cash in).
With only a few exceptions, ML is incredibly simple (there is no AI). The math is simple, the mechanics of evaluating it is simple, the reason it works is simple, and it only really works well if you have absurd amounts of data and CPU time.
Physics is... determining the mathematics you need to know on the fly while discovering and explaining many phenomena. You can spend decades focusing on matrix multiplications and other fairly straightforward trivia to analyze your particle trajectories, and then suddenly, you need to know group theory or some completely different field of math just to understand the basic modelling.
Personally I think the most impressive thing in physics and stats so far is our ability to predict the trajectories of solar system objects far into the future. After some very serious numerical analysis over the past 50 years, we've reached the point where there aren't many improvements we can make, and most of them come from identifying new objects, their position, and mass (data/parameters), and the real argument is about whether the underlying behavior is truly unpredictable even if you have perfect information.
Of course, the last best work in this area was done by Sussman who has been an ML researcher for some time: (https://www.researchgate.net/publication/6039194_Chaotic_Evo...)
As you can see, physicists pretty much invented all the math to do ML whilst solving other problems along the way: https://en.wikipedia.org/wiki/Stability_of_the_Solar_System
in fact many of my friends who were physics people, when I show them the code of a large scale batch training system they wonder why they did physics instead of CS because the math is so unbelivably simple compared ot the tensor path integrals they had to learn in Senior Physics.