I've always been puzzled how it could be possible for a machine to predict stock markets, even on an purely hypothetical basis. To me the idea seems to generate the same kind of paradoxes as time travel or oracles.
I mean, imagine there is a machine that is capable of predicting the markets with 100% reliability. Then soon enough, the machine will acquire a reputation and everyone will follow its advice. The machine says "the price of X will go up", everybody would buy X, so the price of X would indeed go up, but this price surge would have only been initiated by the machine statement and we'd have a self-realizing prophecy. Other situations would be more complicated I suppose, but in every case the machine would have to take into account its own prediction, which would be some kind of a mise en abyme.
In the end just like antic oracles, the machine would have to make cryptic predictions that would make sense only with hindsight.
Try out your thought experiment again with a machine that can predict markets with 55% reliability, or 50.1% reliability . This more likely reflects reality.
If someone had a machine that could predict the stock markets with 100% reliability, or even 55% reliability, do you think they would make those predictions known to the public? Or would they start a hedge fund and become one of the wealthiest individuals of all time?
As far as I can see, making the information available to the public isn't even a choice. There is no market information you can make available to the public that will make everyone richer. The only people who get richer are the ones who act before it becomes public knowledge. When they are done, asset prices reflect the now public knowledge, and there is no longer a profit to take.
What the avocado said, you only need to be better than random chance, even a small bit and you can slowly grow a principal investment over time.
It isn't a whole lot different than an FFT which pickts apart frequency components, HFT algorithms amplify correlation and dampen noise. Then they trade on the correlated signal against the affected securities in an anticipation of the correlated signal's effect.
If correlation was equal to causation they would be 100% effective at making lots of money, but that isn't. So to the extent that the algorithm identifies a correlation that is close enough, it makes money. But it can be crazy like 'sunny days in Paris' is correlated with higher shopper volume in at 'The GAP' and as the sunny days begin to pile up the algorithm buys GAP shares and as the sunny days decline it can sell off GAP shares.
That example we can imagine that people think about buying new clothes when it is sunny but the algorithm doesn't care what the correlation is, just that it has a strong probabilistic link to the changing value of a security. Nothing magical about it.
I've seen this manifest itself in various forms. In some ways, it's like the classic pump and dump scam that used to plague the Internet stock forums back in the late 90's. Tokyo Joe, I think his name was, became infamous for using his track record to front run his mostly illiquid OTC stocks. Because people thought he was this great trader, they would plow into his announced picks, creating a surge of volume in these thinly traded stocks, in which Tokyo Joe would sell out and cause a frantic collapse.
Nowadays, though, you see this sort of behavior in legal and more respectable ways. However, I think the purported results are highly disingenuous. What do I mean? Let's use a well known investment advisory service known as AAII.
AAII has a model shadow stock portfolio with fairly well known and transparent stock selection criteria. However, they often announce stock picks and stock sells to their followers within a day, supposedly, of their actual transaction. I've been studying the behavior of their stock pick/sells announcements, and on average, stocks are trading +5% to -5% (depending on whether it was a buy or sell) within the day following the announcement. This is HUGE for many reasons:
1) They often have multiple transactions on the portfolio in any given year.
2) The performance on the portfolio is boosted by, on average, 5% for any buy or sell. Compounded, this makes a big statistically significant difference on their purported shadow stock portfolio's CAGR and strategy.
3) They do not disclaim this behavior as a caveat emptor.
4) Though their strategy is transparent, their data feed / source is not. No one will be able to replicate their stock picks/sells on their own with their own live data source.
Quant shops look for patterns in the data to try to predict how the stock market moves and codify this in the model. If everybody is looking for the same pattern then the pattern becomes worthless. The patterns are therefore a closely guarded trade secret.
Quant shops have to continually research to detect new patterns to get ahead of the competition for this reason.
Then they also have to account for what other quant shop models are doing. Something something brightest minds of our generation playing algorithm battles. It's sad to me that such pursuits generate massive amounts of money while not benefitting anyone in the slightest.
To my understanding simple regression models worked pretty well the first time they were used. My laymans understanding is that the prediction of a simple model of the system like a stockmarket becomes unusable once there is a feedback loop from the model to the system. I.e. enough users using the same model that the model starts to affect prices.
Nobody can know with 100% certainty what the weather will be like tomorrow but you would be a fool not to look at forecasts. Especially if you are are farmer or are planning a picnic. The mental shift to statistical thinking is something that takes a lot of time and effort. I don't think it is valid to cook up so called paradoxes using arguments from non probabilistic domains.
I mean, imagine there is a machine that is capable of predicting the markets with 100% reliability. Then soon enough, the machine will acquire a reputation and everyone will follow its advice. The machine says "the price of X will go up", everybody would buy X, so the price of X would indeed go up, but this price surge would have only been initiated by the machine statement and we'd have a self-realizing prophecy. Other situations would be more complicated I suppose, but in every case the machine would have to take into account its own prediction, which would be some kind of a mise en abyme.
In the end just like antic oracles, the machine would have to make cryptic predictions that would make sense only with hindsight.