Any company that wants to show benchmarks should, at a minimum show:

    1. The hardware on which the tests were run
    2. The software (OS) on which the tests were run
    3. The design of their benchmarking utility
    4. How much memory/CPU was available for each test (i.e. how much crap is running in the background)
    5. The version of each thing (parser) they are testing
    6. How many trials they ran for each test (no reason for a test like this to not have several hundred or thousand)
    7. The average and standard deviation for each thing (parser) they are comparing
    8. The data ran for the tests. (Some data suites may favour certain parsers over others). Best case: running different kinds of data suites in separate tests, to see if some things (parsers) handle certain kinds of data better than others. (i.e. there might not be a global optimum parser)

Additionally, they should also run the parser on a few passes prior to starting benchmarking to "warm it up". This reduces variance in the runs, especially via things like initialization/memory allocation in the first run.

Without most (or all!) of that kind of information, one can't make a reasonable judgement.

It looks like he ran Sam Soffes' json-benchmark: https://github.com/samsoffes/json-benchmarks

That being said here are the results I got on my iPhone 4 (compiled with -Os) Reading: http://yfrog.com/khul0wsj Writing: http://yfrog.com/h0nqgzej

Interesting, but several of the questions I posed remain, some of I suspect which will explain the significant differences between your measurements and theirs. Except for one: NXJson is exponentially slower at reading for you than they claim it is for them, which may be hard to explain away...

haberman's rule for publishing benchmarks: you don't get to claim that you're >2x faster than your already-highly-optimized competitor without explaining why. If you don't know why, you need to figure out why. Without explaining why, I will assume that it's not an apples-to-apples comparison, or that you made a mistake in your benchmarking.

Not quite. See these tweets from the author of TouchJSON (Spoiler: JSONKit is still faster, TouchJSON still has a better interface):

http://twitter.com/#!/schwa/status/94835100075819008 http://twitter.com/#!/schwa/status/94840915075674112

It's not even plausible. Native binary plist reading is typically about 2x as fast as JSONKit (in my informal benchmarks).

So they claim to be about an order of magnitude faster than binary plists, which would be astonishing if it were true.

Somewhat off-topic, but looking at the readme of JSONKit [1] I found: "At this time, there are no plans to support ARC in JSONKit. Although tenative [sic], it is extremely unlikely that ARC will ever be supported, for many of the same reasons that Mac OS X Garbage Collection is not supported."

Which seems kind of weird, seeing as ARC is going to be the default pretty soon.

[1] https://github.com/johnezang/JSONKit

Not that this matters, though, because ARC and non-ARC code can call each other.

Have you ever looked at the code in JSONKit? It makes perfect sense that it won't support ARC. JSONKit has a lot of small optimizations that deal with allocating memory that interfere with ARC and the assumptions the compiler needs to be able to make for ARC to work.

Anyone benched this against a binary protocol like thrift or protocol buffers?

If JSON parsing is a bottleneck in your application, you're doing it wrong.