It would have been more interesting to see an exam with question like "is the population of x in hardy-weinberg equilibrium", where x is something students should know something (but not everything) about if they've been paying attention. It would be possible, but not easy, for a very well prepared student to do the research and figure it out (or at least write an intelligent answer) in an hour. Some bright students who haven't studied much probably would be able to research quickly and come up with a good answer, especially if they split up the task. Some people will clearly be vastly more valuable to teams than others.

In short, I think the professor here picked a question that tilts too heavily toward collaboration. If this approach is really going to work, you need to induce some reluctance among the very well prepared, who may realize that their friends who spent more time preparing for the non-collaborative physics exam will be getting a relatively free ride.

I guess I just don't think this quite brought out the nastiness that would truly arise in a state of nature - something of course I understand he'd rather avoid!

2. The highest score and thus most insightful and highest quality work, according to the professor's own evaluation, came not from collaboration but from one of the 3 lone wolves.

> Although by conventional test-taking rules, the students were cheating, they actually weren’t in this case.

2: The mob still contributed to the work of the lone wolves, it was absolutely still 'collaboration':

> Although the Wolves listened and contributed to discussions, they preferred their individual variants over the Mob’s joint answer.

Then there seems to be the rather odd point of "the students scored 20% higher than previous midterms", but it is unclear if his previous midterms were the same style - a single open-ended question.

We don't have the actual data but whatever the previous results were we can normalize them so the average score is 50%. We also don't know how many students there are, let's call it 100, so 97 pack students and 3 lone wolves.

We have 20% better results for the pack which received a uniform score. Let's say that's the 70th percentile. (This is a simplification we'll have to make do with until the author publishes the actual score data.)

One lone wolf scored at that 70th point, one below, one above.

The 70th percentile represents the best work that the pack of 97 could agree on. Out of the lone wolves, 2/3 of them did as well as this or better.

It is not hard to imagine that there was better potential work than 70th percentile among at least some of the 97 individuals in the pack than what was chosen through committee, work that will never be manifest because a committee is always a compromise.

It is also not hard to imagine how it is that the average went up given that the half of student population who previously scored in the 0 to 50th percentile are suddenly cast into the 70th percentile simply by choosing to join the pack, even though their contribution, insight and understanding remains exactly the same as before.

It is not hard to imagine a scenario where the failing students who have learned nothing and acquired no advancement in their understanding and yet who just scored 70% instead of 0-50% still have no insight. It is not hard to imagine that the really excellent possible work was suppressed by the groupthink of the crowd. Yet the "average" score is now 20% higher.

We don't even have to consider that this single question essay exam was almost certainly graded in a completely subjective manner and thus the percentages don't mean anything objective at all.

Edit: This is far more adversarial than I intended, and realize it's not a direct response to my parent post (which I thought was saying that this test shouldn't really affect the curve at all, as everyone scored the same, besides the wolves). That being said, I'm always happy to rant against using a strict curve for grading.

My Physics 101 course was taught by a boy-genius that had never taught at university level before. He was very smart and brilliant researcher I will grant but was a lousy teacher. He was very principled and a no curve policy in the syllabus and the first test given had an average score of around 30 out of 100 or so and all but 3 failed (I was #3 and had the equivalent of a D-). The test had no material that was covered in lecture or homework but was derivable from either the textbook or related concepts that the lecture covered indirectly.

He had no intention of changing his stance on that test and we had probably 15% of the class dropout but a week or so after he reversed that and skewed it so the average was a D and lowered its overall weight to final grade. Speculation was he was forced as it was very likely he would have failed 70+% of the class otherwise. It was very instructive to us and I think was a very good lesson as I took things much more seriously from then on and so did most of the class. So I thought that curve was not so bad but will tend to agree a strict curve is a bad thing especially if its not due to an unusual event like this.

You are missing the point here. The process of generating ideas from 27 brains, then using 27 brains to filter them down to a manageable number, then dividing up chores for reference-gathering, writing up, and copying is a far more productive process than 27 people trying to do this separately - they answered a much more open and advanced question, and the ones at the back will have understood much more, than if they had all just sat an exam.

They got better grades because groups of humans outperform individual humans by a huge margin, and that is the lesson of game theory. This exam was an elegant way to demonstrate that.

It wouldn't be the first time that a great idea is shot down because it's too far from everyone's comfort zone. But I'm just guessing.

One need to keep in mind, that "cheating" in school means nothing else than stepping outside arbitrary rule system that is, most of the time, totally opposed to what a honest, rational person would do in real life. Not that it's all bad; education is a training framework.

Since some of the questions were pretty hard we ended up studying really hard together to try and crack them before tests.

Though what we did was somehow against the rules I feel I learned a lot more by studying the questions collaboratively than by studying on my own.

I later found this form of collaboration extremely rewarding and effective during university. We spent many afternoon trying to crack hard problem sets and explaining them to one another.

I've found myself in classes where a group had exams, problem sets and other information from previous years (and solved) and they hid and refused to share those.

It was quite productive to go over all these exams as a group and strive to understand the questions, and particularly to understand the patterns, these were the concepts that lay behind the questions and once you had those you could answer many questions.

These days it isn't hard to find example problems and exams on the internet for your class or similar ones.

So while I'd never say that repeating questions is a good thing, I can't be too hard on folks who do (intentionally or not).

I quite liked the article. It'd be interesting to see if the benefits persisted over multiple groups though. I tend to see engagement increase when rules are changed and students need to figure out how to address the problem in a new way, regardless of whether they are collaborating or not. (anecdotal evidence there only sorry ;P)

If exams are more problems than exercises, and you are able to solve most of them before the exam, then you've probably learned enough and the exam is mostly unnecessary.

Needless to say after I recognised the pattern from the first exam I got perfect scores for the rest of the semester.

The goal shouldn't be “Get a higher grade than my classmates”, it should be “Get the highest grade possible”. And the goal for the teacher should be “Translate each student knowledge data into a numeric value”, which is somewhat orthogonal to the experiment.

Another goal is "get the best possible grade", which some students might not have, but most do, and different students value it differently. But depending on the class, this MIGHT reduce to "get a higher grade than my classmates". Or it might not. If you have a chance (and willingness) to affect your classmates' grades, then knowing whether or not the class is graded on a curve is pretty important.

I love it, I think this pretty well sums up how much certainty there is about what people want out of education :)

The other problem with this approach to "group learning" is that it doesn't take into account freeloading students, i.e. those students that want to do the minimal amount of work for the best possible grade. Anyone that did a group project in college or high school knows that there is always that person who just doesn't care, and is happy to get whatever the group gets.

But in voluntary schools where you pursue your own interests (and not some certification), grades are so ineffectual that they're generally nonexistent. At least in my experience. Tests may corrected, but there isn't a grade. You just see what you did wrong. When you ask the teacher for more detailed feedback on your progress, they don't pull a number out of their butts, like "2.7". That whole grade infrastructure is for another reason.

As for "freeloading students," no one cares. (In these more voluntary forms of education.) It is your choice how much effort you deem appropriate.

During the course of a class, if as much education is given to each individual possible, then at the end of the class what does the actual grade matter? These grades turn out to be a poor proxy for things like earning, ingenuity, creativity, changing-the-world, or for that matter even grokking the materials. The only thing for which they're (maybe) a valuable proxy is 'how well can this student juice the grade of the next course in the line' -- a question whose answer ultimately adds no value to society.

It's unfortunate, but in my department about 80 % of the students go up to med school. The GPA plays a great part in determining just what school will accept you, which in turn has massive impact on your future earnings.

Any college student with the sole goal of learning how to learn & think unfortunately doesn't survive the education system.

Which isn't at all easy. Grades certainly have some correlation to knowledge (certainly not "one" though), but in a perfect world we'd be putting everyone through oral examination and putting their projects through peer review on a constant basis. This is expensive though, in time and money and man power.

It was a lot of fun, and easily the best and most informative experience of my educational career.

I am sure that the essay was good. I am sure that some learned a lot. But how many just road coattails and didn't learn much other than how to copy? In a game like this where everyone stands to gain, and nobody loses, the incentives are to let everyone participate because the cost of riding is free. How many took advantage of that?

Also on a task like this, individual personalities of key figures matter a lot. My guess is that just a few individuals were responsible for a lot of the dynamics.

The whole exercise would have been more interesting - and more competitive - if he had started with a grading scale at the start, same rules, but said that he would rank answers against each other, with multi-way ties being ranked at the median of the group. (So if 5 people tie for 3rd, they will all be scored as 5th.)

This would push cooperation/competition to a new high.

It's not so good for the specific case of animal behavior, in which the whole herd will indeed often prosper or suffer at once.

Education could look very different, and more diverse.

Universities have certain functions in society. If they don't fulfill those functions, they won't get government and corporate funding. Even Salman Khan explains that one important function of schools is to subjugate, fashioning people into obedient cogs. (In his "The One World School House", in the chapter "The Prussian Model".)

That our western civilization morality tends to align with the interests of large-scale cooperative systems is probably no accident. I limit this to western cause I have no clue about any other cultures and I don't want to make an ass of myself.

Of course, this in itself says nothing about the goodness of the system. Cheating a bad system is always an interesting thought experiment. Breaking bad systems and replacing with a new, better one often requires a great degree of 'cheating'. All sorts of interesting things here.

Society doesn't set up rules and systems, they seem to emerge as a consequence of beneficial cooperative behavior towards a 'goal'. When it's no longer beneficial to cooperate towards a goal, over say other strategies like cheating, the rules are overlooked, passed, or changed as quickly as they emerged. The prevalence of War as a way to 'rebalance' resources comes to mind immediately, but so too does the reality that only 2% of people pay use taxes.

To your point about parasitism. Human societies are always in a state of change with respect to sociological and technological concerns, and the success of a society in achieving its 'goals' seems largely dependent on how well it adapts to this change. The parasitism you've described seems to be more a reality of economics than society in general, and though your point seems valid on its face, You wouldn't say that the success companies like Uber, which are cheating parasites by your definition, is really a net negative for the society.

to your point about morality. I believe morality is really orthogonal to this issue and should be considered from different viewpoints in different contexts because humans can hold contradictory points of view and still remain 'functioning'. History holds many examples of individuals who challenge what is 'moral' in a society through cheating, and they are sometimes lauded, sometimes not. I think of the narrative surrounding Rosa Parks on one side ( in support of cheating ) and the story of Cincinnatus ( in support of not cheating ) when considering morality in the context of the cooperation cheating dichotomy.

and you're right, this is a very interesting topic, definitely outside the scope of this small classroom experiment.

Regarding the second point, I really tried to cover this angle with my little bit about 'goodness' of a system, and also vastly over simplified things. Obviously different people in a system are effected by it to different extents, and can play vastly different roles. Many of the statements I was making were with regard to systems with all members are on roughly the same footing. Imagine an honour system for paying for soda, an agreement that everyone on a street clears their own segment of sidewalk of snow, that type of thing. My assumption was that systems and agreements where everyone is more or less on the same footing are generally beneficial to everyone.

Obviously, the situation with regards to taxis and Uber is different. The majority of people are just consumers. They don't really care as long as they are provided the same benefit (reasonably priced, safe, quick, point to point transport). The taxi companies on the other hand are in a completely different position, and in fact, from their position, Uber is definitely a cheating parasite, and a net negative for the taxi companies. I think this falls under my lazy little blurb regarding 'breaking bad systems'.

Cheating has a risk of getting caught long after the moment of cheating and getting caught even decades later can undo a lifetime of otherwise hard work within the rules. Check out the recent scandals in Germany regarding politicians plagiarizing writings during their education.

Don't get caught has an indefinite possible downside to it, you can still get caught even if you weren't caught so far. That's a permanent risk and work hard within the rules has no such downside. Whatever is achieve that way stays achieved.

Strategy #1 (follow rules) only works if you make strategy #2 (cheating) costly. That requires that there are people follow strategy #A of wanting to punish cheaters rather than strategy #B of not punishing cheaters. (From the point of view of someone following #A, someone following #B is cheating, since they are a free rider off of the efforts of people like you.)

The problem is that strategy #A is costly. But following it is more tenable if other people who follow the same strategy actively reward people who demonstrate that they follow strategy #A by outing cheaters. However when people who follow strategy #A become either too common (there are not enough cheaters to catch to reward the vigilance) or too rare (you're overwhelmed by cheaters), then #A becomes not worthwhile, which increases the prevalence of cheating. This proceeds until either there are enough cheaters that vigilance is worthwhile, or until that rule breaks down entirely.

Therefore you can only observe #1 (rule following) being a successful strategy when there are people around following #A (moral outrage at cheating).

In real life people switch between these strategies depending on the rule. For example I get morally outraged at people who cheat in school. I cheat on the speed limit (but within the limits that I think the police will accept).

However when you compare societies, you can clearly see that ones which discourage cheating, have far, far better outcomes than ones that broadly accept it.

In that note, my brother made a recent observation that I like. He's lived both in the USA and in China, and has dealt with people at various levels of society in both countries. His observation is that the broad mass of Americans tend to be relatively moral, but the people at the top in the USA tend to be corrupt. But China is reversed - the average citizen will gladly cheat you if they can, yet the people at the top genuinely want to do what they think is best for the country. (Note, my brother got to personally know many people who are highly placed in the Chinese government while he worked at Lovells and was putting together the Sino-Global Legal Alliance. His opinion of their characters is based on personal interaction, not media accounts.) His personal opinion is that this is going to be a net positive for China going forward.

Your miss-application of 'morality' in the US/China example is blatant racism, and should not be tolerated. It's also completely false, and the worst kind of personal conjecture. Just because you have been socialized from a Judeo-Christian/Greek Moral traditione and Western does not give you the right or the authority to make such a judgment on another culture.

I can't even respond cogently to your arguments on cheating my computer is smoking so hard.

People frequently misunderstand moral relativism. It does not properly mean that you can't make moral judgements. It just means that you can't expect people of other backgrounds to agree with those judgements.

My favorite statement on cultural relativity comes from Charles J Napier, about the Indian custom of Sati, "Be it so. This burning of widows is your custom; prepare the funeral pile. But my nation has also a custom. When men burn women alive we hang them, and confiscate all their property. My carpenters shall therefore erect gibbets on which to hang all concerned when the widow is consumed. Let us all act according to national customs." By British standards, the burning of widows alive was immoral. By Indian standards, failing to do so was immoral. There can be no meeting of minds on the moral question - only people acting as their morals compel them to.

Now let's move to the comment that offended you. My brother has lived in various parts of Taiwan and China for the better part of two decades, interacting with people at all levels from uneducated peasants to the state council. He has had many things to say about Chinese people and culture, some positive, and some not. I regard his observations as observations, not racism.

Let me repeat this observation. His observation is that the average Chinese person is much more likely to try to lie, steal, or take advantage of you than the average American. Not a question of right or wrong, just a fact of daily life in the two countries. But his observation of high Chinese officials - based on personal interaction - is that on the whole they sincerely are trying to make things better for everyone. His (admittedly less direct) observation of elites in the USA, particularly flavored by issues like CEO compensation, the financial crisis, and so on, is that they are out for themselves with little regard for doing what is right in any broader sense.

Use what labels you want for the observation. I still find it interesting.

Let me make an observation. The average Chinese person has much less real 'wealth' than the average American, and when you hold 'wealth' equal, American's are just as likely to lie, cheat and steal, because lying cheating and stealing have little to do with morality below a certain socio-economic level. You just can't generalize a result of poverty to an entire class of people, and then try and make a comparison to another class that doesn't live in poverty.

And if we really want to get down to it. Average Americans have more of their basic needs fulfilled, and so less excuses for 'degraded morality' and yet it is they, and not the 'immoral chinese' who waste more food per capita when people in their own country and the world are starving, it is they who waste more water when over a 1 billion people don't have access to clean drinking water. It is they who through their political power and free-information ( certainly more than the average Chinese person ) choose to directly and indirectly support a government which subjugates not only millions of people directly around the world through its military empire, but also uses its preferential status as the world super-power to drive political and commercial entities in those same countries to commit terrible atrocities on their own people. All for the sake of maintaining the lifestyle of the Average American, who is so moral. and I'm sure if the chinese were afforded the same privileged position they would behave the same as the americans. But you do not recognize this, and feel it necessary to make a false equivalence and assert the moral superiority of one v the other. And it is this moral superiority complex that you seem to have, and which you seem to apply so liberally to Americans, but not Chinese who are evil and bad for stealing and lying and reacting to poverty, nevermind the people in poverty in the US who do the same thing, which makes your observation racist.

As to your selection bias for evaluating elite morality. I would first indicate that the US has a much better and more vigilant news apparatus than China's. And that from a US perspective you haven't been 'informed' on all the bad things the chinese gov't does to its own people. Like how many people died as a direct result of flooding and displacement from the three gorges dam project, how many continue to die for lack of an EPA in beijing and other cities? All people are equally corrupt, regardless of nationality or socio-economic status ( above a certain level, below which 'morality' ceases to be relevant ). The reason why it appears the elite are more so is that their status as elite gives them much greater impact and reach than the average corrupt person.

I don't have a problem with you giving an opinion or observation or even stating your moral position, I have a problem with the assumptions behind it, and the moral superiority you are displaying which I believe to be racist.

I never said that Chinese people are inherently worse people. I understand the effects of poverty, which my brother has also talked a great deal about. There are other possible explanations, such as the fact that China has had to put up with great population density for longer. But for whatever combination of reasons, my brother has experienced a higher rate of petty corruption in daily life in China than he has when he lives in the USA and Canada.

Are his observations accurate? Let's look at corruption in daily life as measured by http://www.transparency.org/cpi2012/results. I see Canada at #9, the USA at #19 and China at #80. (Lower is better.) So according to that he's right, corruption is more of a problem in China on a personal level than in the USA.

On the elites, despite the fact that I said it multiple times, you still have not acknowledged what my brother's basis of judgement is. My brother IS NOT working off of media accounts. He's working from personally knowing a number of them. He'd be the first to acknowledge that horrible things happen with great frequency in China. And with great frequency these things can be traced back to bad decisions by the elites. Who are an unelected group of bureaucrats that maintain power through military force. However he'd immediately come back with an explanation of exactly how huge the problems that they face are, and how rapidly things have been improving. His personal impression is that horrible mistakes notwithstanding, he's not aware of any point in history where so few people have done so much good for so many.

I have no idea how defensible that claim is, but he's a smart guy and he has more background on this topic than I do. And likely than you do. Have YOU ever lived within walking distance of the Premier of China? How many meetings have YOU had with cabinet ministers?

Finally you keep on accusing me of racism. I really don't think that the shoe fits. Yes, I know that it is a common trope to have a racist asshole who says he can't be racist because he's got a friend from group X. So I won't mention my various Chinese friends. And let's ignore the fact that of the half-dozen customers in this coffee shop, I'm the only Caucasian at the moment. But I do have several older half-siblings who are half-Chinese. I live in a neighborhood where many of my neighbors speak Mandarin as a first language. My daughter's best friends are Chinese.

Were I truly racist, some of this should create emotional conflict for me. I assure you, I feel no such conflict. Of course I can't stop you from continuing to accuse me of racism. But I would suggest that continuing to take personal offense at what I have said lets you reject it without having to think about it. Thinking about it might be a worthwhile alternative.

Just because people within a culture may not share judgments about it doesn't prevent a culture from being judged.

I hate it when people try to tell me what thoughts I may or may not try to think.

Hopefully later in life these students will be doing something new and original where nobody else in the world is doing the exact same thing - which means they will naturally end up with nobody to copy off of. If they didn't learn how to figure things out for themselves, they'll end up useless for doing anything new.

That's not a good life lesson and leads to entitlement issues and a society where you can't take any statement of accomplishment at face value.

While it would take a long time to administer these tests for every student it would provide a lot of value for many subjects, while a written test would still be important to determine particular skill sets for certain subjects like math.

Any average student can jam a notecard full of likely facts or cram facts into their head the night before an exam, without learning much. Likewise working as a team to generate the best answer can allow people to receive a high grade without knowing much.

There is also the fact that even a well read student with a good understanding of the subject matter can a fail a test because they don't have a the particular points asked on the test.

My family originated from East Europe and as such most of my family elders have an old school Soviet experience in education. Consequently they have little knowledge and no experience when it comes to modern-day Western education infrastructure. For instance, my parents were genuinely surprised to hear that I wrote written exams and submitted them only to be reviewed at face value - which is the experience of nearly all undergrads I expect. Their experience was that, even in a "straight-forward" subject like Mathematics, they had to first submit a written response to a problem set in an examination type settings - and then upon completion they would have to explain their solution and approach one-on-one with a faculty member who would only then submit a grade.

Obviously this was done to weed out the cheaters who couldn't explain why they had to correct answers to all the questions. But the result can't be ignored that one-on-one oral examinations, while not exactly pragmatic, are very effective!

It will also be an interesting experiment to take a large group of students, split them into groups, and then give out exams to each group. By adding a factor of competitiveness test scores might even go higher. Competition is everywhere in nature.

This can be mitigated if students are working in groups, and group scores are posted instead of individual scores. This way, a group who is scoring low can help and inspire each other to do well and share the pain (shame)

I'm American and we had that in my high school.

In high school astronomy, we could bring in a few pages of notes. I once wrote out the notes in great detail, FORGOT THEM AT HOME, and got the highest score on the test.

In college I learned a lot studying for tests, both multiple choice (geology) and essay (history), where the questions were highly predictable.

I love the test Hilary Putnam announced the first day of every class:

Part 1: Write a suitable question for this class. Part 2: Answer it. You will be graded on both parts.

(Note: The recursive answer of giving him his own question back turns out to be infinitely long.)

I once gave an open book test, saying exam questions would be chosen only from problems I'd assigned for homework. Students still did badly. (Ouch.)

For those who don't want to watch the episode, it's a written exam ninja trainees (genin) need to take to be promoted to ninja squad leader (chuunin). Any team (of 3) that has a single member fail will be failed as a group. If any ninja gets caught cheating they will be given a warning, and after three warnings they will be failed.

SPOILER (no really, the episode is great in this context even if you don't like/watch anime much) The intelligent (but not book-smart) notice the leniency of the anti-cheating-system and the carefully worded instructions to "not get caught", and then find ways to obtain answers from "sources", qualified chuunin who were intentionally planted in the exam hall who knew the answers, or to collaborate with teammates. End plot summary

I think it's important to teach kids not just the answers to problems posed, but also ways to find them. If it's too easy for them to find answers without rigid rules, it probably means that he problems we are posing them are not hard or thought-provoking enough in the first place.

Edited for a missed phrase

It kinda reminds me of CS-whores. The one or two CS girls in each graduation class that regularly used sex-appeal/flirtation to get CS guys to do their homework (very successfully too!).

At some point your no longer testing people's knowledge and creative thinking, but their social skills and ability to manipulate others (and maybe their breast size/size of their 6 pack).

I feel like the professor treated this like a sick little game and enjoyed himself while doing it.

At the end of the day you are competing against your peers either through a curve, or through your raw grades because you're all going into the same fields and are competing for the same jobs with your GPAs. Making a system where cut-throat behavior is encouraged and putting down your pays off is really sad.

Some people, like me, are experiential learners. Others find their best thoughts through discussion and the evolution of ideas. I would have loved to have taken many classes like this. Fortunately, that's what the real world has been like for me - being a member or leading a team to produce the best product we can, either as a team, department or company. As you called it my "pay off" is education. I found mine through methods like this. Not through memorizing facts and methods just to pass a test.

Here the fundamental scoring method is still the test, so an individual's motivation is to do better for himself; the teams success is secondary (and if the test is curved it may be non existent). Free riders aren't punished or penalized and all sorts of outside influences come in where people start trading on money/sex/favors/friendships/etc.

Arguably those outside influences exist in the real world, but why create a system that explicitly has no mechanisms to counteract them? (ie. the prof says you can do anything that is legal and doesn't try to catch "cheaters")

In here, the lecturer specifies that information sharing is not forbidden. Therefore technically this is not "cheating". But it is wonderful to see when you remove the outside limiting factors and let the phenomena occurs in its own volition. Probably this will be one of the courses the participant students will never forget, instead of courses where you "memorize" everything until the last exam.

Kudos to Mr. Nonacs.

Or if he wanted to concentrate on his own work, he could bring some headphones to tune out the horrendous noise coming from outside the classroom where his friends happen to be practicing for their upcoming rock gig.

What's missing is an explanation of why the author thinks this led to an objective evaluation of what individual students learned before test day (if he does indeed think that). That is the point of an exam. Imagine if med students could collaborate on their boards. It would pretty well invalidate the purpose.

Grading on a curve wouldn't be the same game just 'more interesting', it'd be a completely different game, and one that's lost most of the "To win at some games, cooperation is better than competition" aspect.