A lot of comments on this article so far are focused on blindness of the review system, control over publications, etc. which is definitely relevant, but is only half of the story.

The other half is the reasons why papers are popular, and how credit is attributed in papers, grants, and other research.

First there is the question of why papers are published, then why they are cited, and then how peers mentally attribute credit in those cited works. At each step of the way, there are problems: there are biases in why papers are published, why they are cited, and how credit is attributed. A similar process happens with grants.

I agree that all of society is susceptible to this. I think the problem is that in the sciences, we pretend it doesn't happen.

With music, for example, society has a clear understanding of quality and popularity, and the distinction between the two. There might be arguments about the two, such as how strongly related they are, or whether or not quality is even a meaningful attribute of music, but I think most people understand that they are not the same, in one way or another.

With science, though, we act as if they are the same. There's an assumption that, sure, there's some noise, but in the end it all evens out and attribution is correctly given and good ideas rise to the top. No corruption, taking or giving credit inappropriately, no cognitive errors or biases, no nepotism, no etc.

The set of assumptions underlying science reminds me a lot of homo economicus in economics. A set of assumptions that are almost certainly untrue, with significant consequences, but which we tend to ignore out of convenience. (I actually think there should be more skeptically rigorous decision-theoretic/economic analysis of different scientific systems and structures, like game-theoretic analyses of different scenarios as they play out in science.)

One thing on my mind lately is how often people convert from one strategy to another, and how that's patterned and why.

This is the problem: what is the alternative? The pet theory of a madman or sociopath?

The difficult truth is that everyone has an economic theory, whether they admit it or not. The question is, whose theories do we want to pay attention to? People who spend their lives arguing and thinking about it, and trying to find some evidence in support? But who might be a bit isolated from reality? People who work close to the phenomena being explained, but who don't really rigorously explicate or defend any of their ideas, who haven't had to put them on the table, so to speak? Or have massive, critical financial conflicts of interest?

There's also the difficult problem of identifying when any theories have really been tested well. It's not like we can just run randomized controlled designs on whole civilizations--at least, not most of the time for major policies.

I guess I really don't see the alternative to encouraging the standard academic approach to economics, opening it up to public criticism and discussion, and maybe trying them out when it's ethical and feasible.

Sure, lots about classical economic theory is really ridiculous, and the source of a lot of problems, but then you change those things and then move on. It's not any different from physics theory or chemistry theory in that regard.

To not commit an entire society to a single economic theory. Which means getting rid of a lot of the centralized control mechanisms that currently require just such a commitment.

> whose theories do we want to pay attention to?

Since no one has a theory that is good enough to make useful predictions, as far as public policy is concerned, the answer is "none of them".

> There's also the difficult problem of identifying when any theories have really been tested well.

This isn't a difficult problem. You test theories by comparing their predictions to what actually happens. (Note that a "theory" that makes no testable predictions fails this test.) As the article notes, economics does not do well on this test. But that's not a "difficult problem" of how to test economics; it's just that economists, and politicians who want to use economics to justify their pet policies, don't want to admit that the result of the test is that economics doesn't make good predictions.

> It's not any different from physics theory or chemistry theory in that regard.

Yes, it is, because, as you note, you can run controlled experiments in physics and chemistry, but you can't in economics.

Isn't that just another economic theory?

Yes, in the same way we could formulate a medical theory that says "we don't know enough about disease to know whether our treatments are effective, so we aren't going to create a standard diagnostic manual". Prior to the last couple centuries, that would probably have been an improvement.

No, in the sense that, compared to most economic theories, you can't really use this theory to justify a particular group being given access to government power.

I think most of the gains we've gotten from the advent of evidence based medicine is building the taxonomy of medicine so we could reason about it effectively. That only works in a larger sense, small groups toiling with different nomenclatures end up wasting effort.

Much the same way economics is building the taxonomy of economies so we can compare and discuss.

Economics is going to become an entirely different animal in the future as we begin to automatically collect fine grained metrics from our automated economies.

Soon we'll be in the era of evidence based economism, and we'll see a huge increase in the effectiveness of it's utilization.

Not what I was trying to say; I apologize if I was unclear. Rather, my point was that during times past, in many cases it would have been better if doctors had not applied any treatment to patients. Not because doctors were fools, but simply there wasn't enough rigor around assessing the effectiveness of treatments.

So too do I advise this for the field of economics. Not to say that we shouldn't strive for rigor, but until we have it I recommend the "doctors" (i.e. economists) refrain from applying treatments to their "patients" (i.e. the nation via policy prescriptions).

> Soon we'll be in the era of evidence based economism, and we'll see a huge increase in the effectiveness of it's utilization.

An optimistic view that I do not share. I recommend the talk "Science, Knowledge, and Freedom" by Jim Manzi[0]. In short, observations in the "soft sciences" of psychology, sociology, and economics do not generalize across different times and cultures in the way that observations about physics, chemistry, and biology do.

[0]https://www.youtube.com/watch?v=N4c89SJIC-M

In reality it's more like "psuedo-scientific mismanagement" vs. "mostly civilized confusion". Neither of these scenarios is particularly appealing, but I'd rather take the confusion.

Regarding economics, there are some fundamental assumptions that are often criticized.

For instance, in the mid 1700s, Charles Townsend asserted the fundamental laws of economics were these: 1,) Man's want, wishes, needs and desires are infinite. 2.) Resources are finite.

But numerous anthropologists have criticized these assumptions. Margaret Mead didn't see the tendency towards infinite consumption among the tribes she studied. And see Marshall Sahlins book Stone Age Economics -- he doesn't see any evidence of infinite desires among the tribes he studies. Thus it appears Townsend's assumptions are fit only for a society in the middle of the Industrial Revolution. So his "laws" are not "laws" in the same way that Issac Newton could describe "Laws Of Motion". Rather, Townsend's laws were a convenient assumption for a certain kind of society, but then the question comes up constantly, are we still the same kind of society?

Another example, Joseph Schumpeter starts one of his books (I think Das wesen und der hauptinhalt der theoretischen nationalökonomie, 1908) "The social process is one indivisble whole! However, to make useful remarks regarding production and consumption, it is reasonable to pluck from experience those transactions that are altogether economic in nature."

There are many people who strongly agree with the first sentence and therefore they can not agree with the second sentence.

I could go on, but I don't have the time.

If you go looking, you'll find many examples. In physics, no one doubts that matter and energy are real things, but in economics, it is the most fundamental assumptions of the discipline that have always been under attack.

In spite of such major deficiencies, economic science was very lucky and promoted the free market, which was in many ways an amazing success. As a consequence economists got the mistaken idea their science was basically sound, and never set about thinking out its basic assumptions.

That's the thing, it has proven really, really hard to "change those things". I'm looking at the latest (still current?) economic/financial crisis. Lots of "economic theory tells us this should never happen"-things did in fact happen, and consistently so (there was not just a random event that you could blame in the theory not being 100% there). It matters because policy decisions are still taken based on theories that have proved themselves wrong, or at least those theories were not able to "explain" (for a lack of a better word) how our present-day economy works. Those policy decisions affect (some of them negatively) the lives of millions, even hundreds of millions.

At least when a theoretical physicist is wrong in his/her assessment nothing that bad can happen to the outside world, in the great scheme of things. But when an economist is wrong, but his theory is nevertheless taken into consideration and acted upon, then the damage can be quite substantial, its effect measurable in decades.

I don't know what needs to be done. I'm not ditching economic theory entirely. For example the Chinese authorities' current push to stop Chinese billionaires' money moving outside the country reminded me of Jean-Baptiste Say's explanation of how no Government can put a stop to the flow of currency that wants to escape a certain jurisdiction, a phenomenon that he wrote about ~200 years ago and which still seems to be in effect, almost like a "law" of economics. And there are still other "basic" economic truths/laws that have seemed to keep their relevance over the centuries. But, AFAIK, almost all of those "laws" were common sense, so to speak, they didn't involve mathematiac equations with second derivatives and the like (for example there's no "second derivative" equation with which to model the "lack of trust" in financial transactions, which once it sets in you can bet will bring any financial market down pretty fast).

So what I'm trying to say in a convoluted way is that economics should return to basics, ditch most of the mathematics with which it has become enamored and try to be a little more on the "social science"-side of things: more observation of humans and their acts, less abstract computations.

An interesting article (that was posted to HN a while back) that discusses this is "How economists rode maths to become our era’s astrologers": https://aeon.co/essays/how-economists-rode-maths-to-become-o...

Though it's been posted a few times, there hasn't been that much discussion on HN: https://hn.algolia.com/?query=https:%2F%2Faeon.co%2Fessays%2...

What economic theory said which things should never happen exactly?

I'm not aware of an economic theory that says mispricing should be impossible, especially for a price set in advance.

> Trillions of euro-denominated deposits stood at bellow 0% for more than a year.

As far as I can tell this is referring to the an ECB policy for deposits of other banks to the ECB. This is not a market price; the ECB is actually trying to discourage its bank deposits. No contradiction of economic theory here.

> Real wages are continuously going down even though the unemployment figures are pretty damn good.

I'm not aware of real wages going down "continuously". As far as I can tell it's remarkably flat. This, however, is closer to something that actually doesn't seem to make much sense at first glance according to economic theory.

FYI I tried googling all of these things using terms you used and mostly got dubious zerohedge articles near the top.

That ECB policy was indeed a price, don't know what they want to "discourage" or not, fact is that if you wanted to deposit money to the ECB you had to pay them for the privilege. AFAIK since capitalism set in properly (about 200 years ago) it has almost been the case that you were supposed to receive money as interest when depositing it somewhere (to a king's vault, to ECB, it didn't matter).

Am on mobile, too lazy to look for the real-wage charts. I had just seen one in the FT detailing its evolution for the UK since 2005 or so, with only 3 years out of those 11-12 seeing real wage increases. I used the same source for commenting on Argentina's debt, i.e. last Friday's Financial Times. I used to read zerohedge from time to time, but I don't like their layout, I've mostly stuck with the FT and the Economist for my economy and financial-related info (there was also an interesting economics-related blog under the economist.com domain).

http://econlog.econlib.org/archives/2015/09/how_the_subprim....

A large number of economists (in academia, public institutions, and private finance firms), all across the ideological spectrum (Austrians, Keynesians, and every other flavor) did see it happening.

People didn't respond to then, probably because the existence of a bubble isn't a problem, as long as you can delude yourself to thinking you'll be able to time the market so you won't be holding the ball when it pops.

> Instead the economics profession, with a few exceptions like Dean Baker, told the world that things were going just fine.

Baker was far from the only prominent economist pointing to a bubble. In fact, economists were warning about the housing bubble and the fact that it would have to burst before the first dot-com bubble burst (or even expanded) back as far as the mid-1990s. That may actually be the real problem: the warnings had been around for so long no one took them seriously any more; as controversial as identifying a bubble can be, it's a lot easier to identify it than time when it will pop ,and the longer it is pointed to without popping, the more likely people are to convince themselves it's just a permanent feature of the market and not a bubble that will pop.

In a strange way, that's almost step down: A strong correlation between what economists say and what happens -- even a negative correlation -- suggests that theory is somehow catching up to reality.

In contrast, a weaker correlation -- even if positive -- implies that there's still a lot more problems to shake out.

To me it seems like the paper isn't really offering a fundamentally new explanation of how these traits arise, it's just providing a sophisticated analysis of the dynamics.

In other words, the field of evolutionary biology today has its Newton's laws and Maxwell equations, but we're missing anything like the Standard Model. The authors of this work have been at this for a number of years. A sophisticated analysis of the dynamics is a key and very important step in beginning to develop a "grand unified theory" of evolution.

MIT Tech Review has often news about fascinating research, but one has to adjust for some amount of unnecessary hype.

The piece is kind of making a basic fundamental mistake in measurement, assuming that all variability is meaningful variability.

There are ways of making the argument they're trying to make, but they're not doing that.

Also, sometimes a single overall score is useful. A better analogy than the cockpit analogy they use is clothing sizing. Yes, tailored shirts, based on detailed measurements of all your body parts, fit awesome, but for many people, small, medium, large, x-large, and so forth suffice.

I think there's a lesson here about reinventing the wheel.

I appreciate the goals of the company and wish them the best, but they need a psychometrician or assessment psychologist on board.

We aren't trying to make a rigorous statement here -- we're trying to draw attention to the fact that the most common metrics do not give much insight into what a student has actually shown mastery of. This is especially important when you consider that the weightings of particular questions are often fairly arbitrary.

I certainly agree that all variability is not meaningful variability, but I'd push back a bit and say that there's meaningful variability in what's shown here. We'll go into more depth and hopefully have something interesting to report.

I've also seen a fair number of comments stating that this is not a surprising result. I'd agree (if you've thought about it), but if you look at what's happening in practice, it's clear that either many people would be surprised by this, or are at least unable to act on it. We're hoping to help with the latter.

If by mentioning free response, you mean students are unlikely to guess the correct answer, even when they don't know it, it's a 2 parameter IRT rather than 3.

Best of luck! :)

I am generally of the philosophy that you should trust no one to do the right thing, but these cases seem to be overlooking the obvious that the phone companies are fucking up on security.

I'm starting to worry about similar weak process security on the part of the IRS and Social Security. You can theoretically opt out of using a cell phone, but it's far harder to opt out of government programs that are forced on you with the threat of state force.

However, you're also bringing up an issue that seems really neglected in public discussion, which is the role of municipal infrastructure and policy.

To me there's three things:

1. The importance of net neutrality to the working of the internet, at a theoretical level.

2. The lack of competition.

3. The obligation of companies to the public because of advantages they've been granted.

These discussions always proceed like big ISPs own the land that cables are going over, etc. when that's anything but the case. There's things like the telephones you mention, plus right-of-way laws.

Why the EFF doesn't publish a huge map showing land used by ISPs that's not owned by them I don't understand.

It outrages me to see these ISPs being given right of way, and then having them turn around and act like they don't owe anything to the public, when in fact they wouldn't exist without the public bending over backward to give them all sorts of privileges. They're acting like parasites.

To me I might take arguments against net neutrality more seriously if those arguing against net neutrality would:

1. Lobby for the rights of municipalities and other institutions to set up their own ISPs and networks. No laws against public ISPs.

2. Lobby for ISPs to be held responsible for any and all criminally or civilly infringing content going over their networks. Want to preferentially treat packets? Then you should scan for child porn too. If not, you're implicitly supporting it.

3. Lobby for the elimination of right-of-way. Let municipalities, state and federal governments, and private landholders charge ISPs whatever they want for access over their land.

Of course, ISPs would scream bloody murder and argue such things are ridiculous, but I would argue that they're not at all unreasonable given what the government is asking for in terms of net neutrality. They want to have their cake and eat it too.

Likely because it wouldn't show anything interesting. State and local governments may only "manage the public rights-of-way or to require fair and reasonable compensation from telecommunications providers, on a competitively neutral and nondiscriminatory basis, for use of public rights-of-way on a nondiscriminatory basis, if the compensation required is publicly disclosed by such government." https://www.law.cornell.edu/uscode/text/47/253. In practice, rights of way are poles and conduits, which are rented to all companies at published, non-discriminatory rates: https://www.law.cornell.edu/uscode/text/47/224. The idea that companies are sitting on all these free rights of way they don't own is largely false.

I don't know what's causing the hold up in Nashville. The lawsuits don't stop Google from deploying, only from relying on the city's "one-touch make ready" ordinance which would allow bypassing the FCC process. But Google deployed in Kansas City, etc. without the expedited process, so that doesn't explain the delay. And Google Fiber's deployment in newer Fiber cities like Atlanta has been slow even though they face no litigation there. I suspect it all has to do with the fact that Google Fiber only has like 1,500 employees and recently cut 10% of its workforce.

I'm very good at identifying needs, in the sense of "here's a fundamental problem, and here's some ways of addressing that problem."

However, I'm not very good at identifying ways of turning that into a profitable enterprise. Often when I think of problems and solutions, it's because others are neglecting something, and aren't even aware of the problem, so there's no motivation to pay for any solutions. That is, you'd be selling something that people don't want because they aren't even aware of the looming problem or risk they have. Later on, sure, when things fall apart, everyone wants the solution I had in mind, but at that point it's obvious and there's too much competition.

My other problem I run into I get too absorbed in my own interests and am not really motivated enough by the profitability of something, even when I know I should be more motivated by it. So here's two ideas, A and B. I'm very interested in A and see it as important, but maybe not so profitable. B is less interesting and maybe less important but more profitable. I subconsciously tend to gravitate toward A, to the thing that I see as interesting and important, but that might not garner a lot of recognition or compensation in the short-term.

I think so far I've been kind of unstrategic about where to go in life, and people have just seen me as smart and valuable enough to have around to solve problems. That's gotten me fairly far, but I've reached a point where maybe I need to be more entrepreneurial.

I've also seen enough things in my life to know that there's a ton of unpredictable social dynamics that go into these ventures, and I'm kind of burned out. Fads, corruption, etc.

What's stopping me? I think it's mostly burnout and disillusionment.

My sense is that the level of understanding of how human cognition works mathematically speaking is roughly similar to our understanding of how DL works in computer science, in the sense that if you asked a cognitive neuroscientist or psychologist how someone classifies cats versus non-cats, you'd get an answer that would seem pretty similar to what you'd get from a computer scientist. The behavioral scientist might go into a lot more detail about certain issues, but that's because the biology is so entertwined.

However, I'd also argue that we really don't know much about how human (or any animal) cognition works, and I'd also argue that our understanding of DL is fairly poor, in that a lot of it is tinkering and seeing what happens, without a deep understanding of why it works. There isn't a theory of DL in the same way that there's a Martin-Lof theory of randomness, or a Kolmogorov theory of algorithmic complexity, or a Fisherian model of inference.

Also, the sort of tasks currently involved in AI research is a tiny subset of what you encounter in neuroscience and psychology. Most of what is a hot topic in comp sci would basically be classified as perceptual tasks in human behavioral science, maybe at a slightly higher level, and maybe motor control. That leaves things like conscious versus nonconscious processing, reasoning, the role of emotion in decision-making, uncertainty valuation, creativity, etc. etc. etc.

I agree that the processing power is an issue but it's only part of the puzzle.

One thing that illustrates the complexity of the issues involved, and how we've only begun to scratch the surface, is the article's assertion that comp sci should borrow more the idea of sparse representations from cognitive neuroscience. I thought that was interesting, because in a lot of ways, one of the major trends in the last ten years in human neuroscience is away from this "sparseness" idea. It was a common assumption maybe 15 years ago, but now people routinely get excoriated for invoking that idea. The current paradigm is one more where a lot of pathways/circuits are being recruited simultaneously. Statements like "you might use 10,000 neurons of which 100 are active" would lead to ridicule. The intuitive way of explaining the problem is that even while your brain is trying to decide if you're perceiving a cat versus something else, it's also processing the consequences of that decision along about 10 different dimensions, the implications for the rest of the stimuli coming in, along with a number of other things we just don't understand.