No. All the textbooks know that polynomials of high degree are numerically dangerous and you need to be careful when handling them.
The articles examples only work because the interval 0 to 1 (or -1 to 1) were chosen. For whatever reason the author does not point that out or even acknowledges the fact that had he chosen a larger interval the limitations of floating point arithmetic would have ruined the argument he was trying to make.
10^100 is a very large number and numerically difficult to treat. For whatever reason the author pretends this is not a valid reason to be cautious about high degree polynomials.
This means that when using polynomial features, the data must be normalized to lie in an interval. It can be done using min-max scaling, computing empirical quantiles, or passing the feature through a sigmoid. But we should avoid the use of polynomials on raw un-normalized features.
This paragraph has nothing to do with numerics. It is about the fact that continuous functions can not be approximated globally by polynomials. So you need to restrict to intervals for reasons of mathematical theory. This is totally unrelated to the numerical issues, which are nowhere even acknowledged.
All polynomials "work" globally. That some polynomials form an orthonormal basis over certain intervals is essentially irrelevant.
The author does not address the single most important reason why high degrees of polynomials are dangerous. Which is pretty insane to be honest, obviously to be honest you have to at least mention why people tell you to be cautious about high degree polynomials AND point out why your examples circumvent the problem. Anything else is extremely dishonest and misleading.
Actually they aren't. You never compute high powers of the argument when working with specialized bases.
You use the recursive formula that both the Bernstein basis and the orthogonal polynomial bases are endowed with. This is implemented in numpy, so you don't have to do anything yourself. Just call, for example, np.polynomial.legendre.legvander to get the features for the Legendre basis.
And a basis orthogonal over [-1,1] is easily made orthogonal over arbitrary interval. Take p_i to be the i-th legendre polynomial, then the basis composed of
q_i(x)=p_i(2(x-a)/(b-a)-1)
is orthogonal over [a,b]. Each q_i is itself a polynomial of degree i, but you never use its coefficients explicitly.
There is an entire library for computing with polynomial apptoximants of functions over arbitrary intervala using orthogonal polynomials - Chebfun. The entire scientific and spectral differential equations community knows there are no numerical issues working with high degree polynomials over arbitrary intervals.
>The entire scientific and spectral differential equations community knows there are no numerical issues working with high degree polynomials over arbitrary intervals.
This is totally wrong. Of course there are enormous numerical problems with high degree polynomials. Computing large powers of large numbers is enormously unstable and needs to be avoided under basically all circumstances, that is what makes dealing with those polynomials difficult and cautioning people against this is obvious correct.
What you described are the ways to deal with those problems. But this isn't what the article does. My problem with the article is the following:
- Does not mention the most important issue with high degree polynomials, namely their numerical instability.
- Gives examples, but does not explain why they circumvent the numerical problems at all. The most important choice made (the interval of 0 to 1) is portrayed as essentially arbitrary, not an absolute necessity.
- Concludes that there are no problems with high degree polynomials, based on the fact that the experiments worked. Not based on the fact that the actual issues were circumvented, leaving the reader with a totally wrong impression of the issue.
This is terrible scholarship and makes for a terrible article. Not acknowledging the issue is a terrible thing to do and not explain why seemingly arbitrary choices are extremely important is another terrible thing to do. The whole article fails at actually portraying the issue at all.
To be clear, I am not saying that this approximation does not work or that with appropriate scaling and the right polynomials these issues can't be mostly circumvented. Or that high degree polynomials "in general" are incalculable. I am saying that this article completely fails to say what the issue is and why the examples work.
I believe the author assumes that it's clear to the reader that there is a distinction between how a mathematical object is defined, and how it's computationally used. A polynomial can be defined as a power series, but it's not how they are computationally used. In this sense, the author was mistaken.
But it's not that the problems are "circumvented", in the sense that it's a kind of a hack or a patch, but they are solved, in the sense that there is a systematic way to correctly compute with polynomials.
>But it's not that the problems are "circumvented", in the sense that it's a kind of a hack or a patch, but they are solved, in the sense that there is a systematic way to correctly compute with polynomials.
But the author does not even acknowledge that there is a wrong way. He claims that it is just a "MYTH" that polynomials have huge problems.
Read the article, nowhere does the author acknowledge that these numerical issues exist at all. Nowhere does the author talk about why the specific methods he uses work or even acknowledge that had he used a naive approach and written the polynomials as a power series over a large interval everything would have fallen apart.
Surely the "MYTH" that high degree polynomials are dangerous is not a myth. Concrete examples where naive approaches fail are trivially easy to find.
Again. I think you are missing the point entirely. I am not saying that these Fourier type projections are "wrong" or "bad" or "numerically unstable" or that you can't write an article about those or that high degree polynomials are impossible to work with.
I am saying that if you claim that it is a "MYTH" that high degree polynomials are dangerous, you should mention why people think that is the case and why your method works. Everything else seems totally disingenuous.
> Computing large powers of large numbers is enormously unstable and needs to be avoided under basically all circumstances, that is what makes dealing with those polynomials difficult and cautioning people against this is obvious correct
But we don’t compute large powers of large numbers? Chebyshev is on [-1, 1]. Your large powers go to zero. And your coefficients almost always decrease as the degree goes up. Then to top it off, you usually compute the sum of your terms in descending order, so that the biggest ones are added last.
"To be clear, I am not saying that this approximation does not work or that with appropriate scaling and the right polynomials these issues can't be mostly circumvented. Or that high degree polynomials "in general" are incalculable. I am saying that this article completely fails to say what the issue is and why the examples work."
Neural network training is harder when the input range is allowed to deviate from [-1, 1]. The only reason why it sometimes works for neural networks is because the first layer has a chance to normalize it.
The articles examples only work because the interval 0 to 1 (or -1 to 1) were chosen. For whatever reason the author does not point that out or even acknowledges the fact that had he chosen a larger interval the limitations of floating point arithmetic would have ruined the argument he was trying to make.
10^100 is a very large number and numerically difficult to treat. For whatever reason the author pretends this is not a valid reason to be cautious about high degree polynomials.