@ Euan. Did this for about 20 mins - pretty straight forward. Every little helps. - Graham Steel

I did a few too - Nils Reinton

Thank guys! It's a bit of a slog but is really useful info. - Euan

I did 10 or 12. In some places there are repeated comments. - pn

Did a couple -- found one where an author had reported on a journal club dissection of her paper. I called it a "comment from author" since that's who submitted it, but it's also a journal club. Any chance we could pick more than one option at a time? - Bill Hooker

Liked so I will remember to do this when I have a moment - Shirley Wu

+1 "Any chance we could pick more than one option at a time?" Agreed, Bill. - Graham Steel

This looks interesting... Will do some more tomorrow when I have some more time. - Ricardo Vidal

+1 for >1 option. - Chris Cotsapas

I gave a couple of minutes to it this afternoon. Maybe more tomorrow. - Jill O'Neill

Going through a couple now. Recognizing some names. Multiple categories would be better, if it's doable within your analysis. Also there might need to be a category for "discussion among commenters". - Mr. Gunn

bumping this up for visibility - please help Euan out on this. - Bora Zivkovic

Bookmarked , will have a look during this week. - Pedro Beltrao

Doing it as we speak - Deepak Singh

Posted a related thing to my blog (http://phylogenomics.blogspot.com/2009...) and suggested people take your survey too. - Jonathan Eisen

How should we mark comments from authors replying to direct criticism ? are these just comments from authors as well? - Pedro Beltrao

@Pedro yeah, any type of author response is a comment from author rather than anything else. - Euan

I would update the app to allow multiple categories - I think on balance it's a better idea - but I'd have to dump all of the work done so far, so it's a no go. :( I'll try and put up better guidance and a 'annotators discuss' box. - Euan

Incidentally there are now ~ 700 annotations, which is fantastic, thanks everybody! About a third of the way there. - Euan

@Jonathan and thanks for the blog mention! - Euan

Great crowdsourcing idea. Some commenters were familiar (Bora Zivkovic, Graham Steel, Björn Brehms). - Martin Fenner

I submitted this to the science subreddit (http://www.reddit.com/r...) and it generated quite some interest and comments there. Two I liked were to submit this as a HIT to Mechanical Turk, and a seperate category for people pointing out typos. - Jeroen Van Goey

I think I've seen the same comments come up twice. Are you oversampling and taking the consensus ranking? - Mr. Gunn

I've also seen some cases where the author posted the reviewers anonymous comments as a comment. I think that's a useful practice. - Mr. Gunn

@Mr Gunn re: oversampling: yes. - Euan

Good idea - I've done a couple, will do more as time permits. I, too, saw familiar "faces" (I'm looking at you, Deepak!) - Allyson Lister

I've seen a case where a person seemed to be trying to put a trackback in the comment field (they blogged about the article). Could only put it into "other"... - Allyson Lister

OK I've pitched in and done a few too :) - Daniel Swan

Thanks guys! Almost got complete coverage now (waiting for there to be more than one decision on some comments). - Euan

@ Euan, is it possible for someone to only do abstracts in a particular topic area (e.g. Comp bio)? A colleague is asking. Might not matter at this point since it sounds like you're almost there - Shirley Wu

@Shirley sadly not as the articles aren't categorized inside PLoS ONE (I don't think). - Euan

Maybe it would be possible to come up with some silly geeky games for scientists that would help publishers tag abstracts or images. "name that cell type" :). Using the same strategies of Luis von Ahn's games. - Pedro Beltrao

Heh, it is fun... - Anders Norgaard

Google does that sort of labeling game on their images. Not a game per se, but basically tagging images. Crowdsourcing. - Ricardo Vidal

@Ricardo I think the Google Image labeling was inspired on those crowdsourcing games from Luis von Ahn. The problem with Mechanical Turk is that lack of scientific background of the users. I am not sure that even just this task of sorting comments according to the rules Euan set up would work over there. - Pedro Beltrao