For each language, I would have two experts in that language. Call them P and D: programmer and documenter.
They would not be permitted to communicate.
P would be required to write strictly code, without a single comment or a shred of documentation.
D would not be permitted to view the problem description: only the input data (along with documentation describing what the data means) and the code written by P.
D would have to determine what the code is doing and specify the problem being solved, relying on his or her expertise in the implementation language to unravel how the code is treating the input data, and hence what is its meaning.
Then evaluate the time spent by P, the time spent by D, the size the code, and the degree to which the code actually solves the problem and the degree to which D's documentation actually reverse-engineers the specification not revealed to D.
Then we could say meaningful things: like a given solution is very small and was developed quickly, but it took great effort to unravel what it's doing. That's not bad: there are situations when that is acceptable, namely the development of one-off solutions that will never need to be understood or maintained, and which are needed ASAP. Situations would also reveal themselves when the program takes long, and isn't particularly short, but is a breeze for D to understand. Usually we would prefer that situation in permanent production code.
The all round winner would be: small code written by P in a short amount of time, quickly reverse-engineered by D to recover the spec. (With bonus points for time performance, small memory use, etc).
There is also a degree of realism in this setup. How often are we called upon to understand some code which is utterly undocumented? That's when we feel the differences in languages, programming approaches and styles.
Tradeoffs in speed vs maintainability could be the intentional result of choices made by P though. What instructions would they be given regarding how to handle those tradeoffs?
For each language, I would have two experts in that language. Call them P and D: programmer and documenter.
They would not be permitted to communicate.
P would be required to write strictly code, without a single comment or a shred of documentation.
D would not be permitted to view the problem description: only the input data (along with documentation describing what the data means) and the code written by P.
D would have to determine what the code is doing and specify the problem being solved, relying on his or her expertise in the implementation language to unravel how the code is treating the input data, and hence what is its meaning.
Then evaluate the time spent by P, the time spent by D, the size the code, and the degree to which the code actually solves the problem and the degree to which D's documentation actually reverse-engineers the specification not revealed to D.
Then we could say meaningful things: like a given solution is very small and was developed quickly, but it took great effort to unravel what it's doing. That's not bad: there are situations when that is acceptable, namely the development of one-off solutions that will never need to be understood or maintained, and which are needed ASAP. Situations would also reveal themselves when the program takes long, and isn't particularly short, but is a breeze for D to understand. Usually we would prefer that situation in permanent production code.
The all round winner would be: small code written by P in a short amount of time, quickly reverse-engineered by D to recover the spec. (With bonus points for time performance, small memory use, etc).
There is also a degree of realism in this setup. How often are we called upon to understand some code which is utterly undocumented? That's when we feel the differences in languages, programming approaches and styles.