1. It was funded by Duolingo but carried out by an outside team of academics. I can't judge how much this impacted their analysis.
2. No control group, which isn't very promising for the rest of the methodology.
3. More time spent using Duolingo did seem to result in higher improvement
4. Lot of dropouts, some participants excluded for taking up courses. They tried to control for 'outside resources' like watching movies in Spanish but it's unclear if that problem was really solved.
5. Novices of the language learned the most. Makes sense, as Duolingo focuses a lot of vocabulary and basic sentence construction
Personally I'm using Duoling to learn Italian, and I find that I am definitely learning, especially in terms of conversational phrases and vocabulary.
However, I've learned a few other languages by immersion, and this is going much, much slower. I'm 99% sure I would learn more by moving in with an Italian family or by consuming all my media in Italian, instead of Duolingo. For now, Duolingo is a nice compromise, and I accept that I'll learn a limited amount from it.
I think the correct conclusion from that study is not that Duolingo is good, but rather that it is not quite as bad as typical college language courses which are truly terrible.
https://static.duolingo.com/s3/DuolingoReport_Final.pdf
A few notes about it though:
1. It was funded by Duolingo but carried out by an outside team of academics. I can't judge how much this impacted their analysis.
2. No control group, which isn't very promising for the rest of the methodology.
3. More time spent using Duolingo did seem to result in higher improvement
4. Lot of dropouts, some participants excluded for taking up courses. They tried to control for 'outside resources' like watching movies in Spanish but it's unclear if that problem was really solved.
5. Novices of the language learned the most. Makes sense, as Duolingo focuses a lot of vocabulary and basic sentence construction
Personally I'm using Duoling to learn Italian, and I find that I am definitely learning, especially in terms of conversational phrases and vocabulary.
However, I've learned a few other languages by immersion, and this is going much, much slower. I'm 99% sure I would learn more by moving in with an Italian family or by consuming all my media in Italian, instead of Duolingo. For now, Duolingo is a nice compromise, and I accept that I'll learn a limited amount from it.