I don't know if this is related but I work on a project with a language parser and so a tokenizer and one issue we have is the parser/tokenier was not designed to work on incomplete code so it's not useful for auto complete.

That property of how good the design is for helping write and debug (useful error messages) seems like yet another metric that could be used

I also don't know if tokenizers can be divorced from parsers. Some languages have constructs that require context that comes from the parser