Rockin' idea. I was just looking for something like this to do some quick-and-dirty market research for my startup.
I don't have much experience with mturk, but my one concern would be that respondents may simply click randomly rather than honestly answering the question; do you have any data on how honest and accurate your responses are?
With any survey you always have the possibility of disingenuous answers, but we require them to put a comment/rationale for their answers, which is where the most interesting data lies. Here are some examples: http://pickfu.com/IOHSK8http://pickfu.com/JWQFLA
Our respondents don't get paid unless they actually explain their answers - which both reduces false responses and provides unique value and insight to the service.
I still think doesn't help. I liked the idea and did the turk for one of your questions (this was my first time doing the turk, btw). I was asked to visit a myspace webpage, listen entirely to 2 songs and say which one I like better and why. All this for 3 cents? I did it, cause I am a good fellow but I have hard time thinking of people who are willing to spend 10 minutes of their time for 3 cents.
the FAQ example is a bad example: I can always answer "yes" and then supply as reason "every questions I had was answered". No need to read the FAQ page, of course.
I think if you want to insert any kind of validation system you would have to count the time it took for the turk to answer your question and make sure we are in that range for every hit.
As noted before, it's tough to know that 100% of the responses are truthful on any survey. With PickFu, you can read the explanations to get a feel as to which respondents put some thought in.
I'm sorry you had a rough experience with that specific music-listening job - we've never had a more time-consuming question submitted. Most are in the order of "which site do you prefer?", like this current one: http://pickfu.com/AGSIBT. These get answered far faster by the MTurk workers.
You're basically going to get the same response quality as Amazon Turk. We use them a lot, but the quality is pretty average so there's still some amount of manual work or quality checking needed.
What would be truly amazing if this service could work with non-mechanical turkers...
I don't have much experience with mturk, but my one concern would be that respondents may simply click randomly rather than honestly answering the question; do you have any data on how honest and accurate your responses are?