The same criticism probably applies to online teaching evaluations done on 1-5 scales. - Dan Gezelter

More sites should incorporate "Pick which is better" style ratings like on http://thecutest.info - Ryan Moulton

I'm enjoying Mr. Crotty's series of "How useful is X, really?" where X=any innovation in publishing occurring this decade. Can't wait for the surely upcoming "Here's my proposed solutions" series. - Mr. Gunn

Seems the key thing is that only ~ 5% of articles are rated. So only people with enthusiasm for the system participate and they tend to be enthusiastic about the content? - Neil Saunders

and, if you buy his line, the enthusiasm of that 5% makes their input untrustworthy. There certainly couldn't be people just using the tools to get a job done and not being advocates, right? - Mr. Gunn