The problem with measuring the accuracy of juries, is what do you measure them against? Whether the person is really guilty? In the general case, you can't actually know that.
Even for people who are wrongfully convicted, we can't always say the jury made the wrong call. In many of those cases, the conviction is overturned due to new evidence being discovered, or because the police or prosecution wrongly hid exculpatory evidence from the jury, or because the defendant's lawyer was grossly incompetent – in those cases, it is entirely possible the jury made the right call given the evidence actually presented to them, even if it turned out ultimately to be the wrong one.
> The problem with measuring the accuracy of juries, is what do you measure them against? Whether the person is really guilty? In the general case, you can't actually know that.
Yes, in the general case you can't know that. That's why you need to construct artificial cases where you do know.
Very similar to getting labelled training data for your machine learning.
For this hypothetical comparison process, I imagine that real cases could be used and re-presented with actors in place of the defendants and witnesses?
Yes, I'm sure you can design an experiment which shows different jury structures will produce different decisions. But how do you know which of them makes the better decisions?
Even for people who are wrongfully convicted, we can't always say the jury made the wrong call. In many of those cases, the conviction is overturned due to new evidence being discovered, or because the police or prosecution wrongly hid exculpatory evidence from the jury, or because the defendant's lawyer was grossly incompetent – in those cases, it is entirely possible the jury made the right call given the evidence actually presented to them, even if it turned out ultimately to be the wrong one.