Last paragraph seems odd: "yes, I have information and insight that could help tenure committees etc make better decisions; no, I won't make it available". I don't understand the connection between that decision and the anonymous nature of peer review. - Bill Hooker

The P2P post author suggested that factoring in review quality along with publication/citation could provide a better measure of an individual scientist. As an editor, I have exceptional knowledge of each referee's reviewing caliber and prowess. But since the review process in anonymous, there is no way to connect good reviews with individual scientists. So until the review process is open, I think attempting to assess peer review as a means to value an individual scientist is silly. - Noah Gray

So tenure and faculty search committees will have to work to determine the worth of their scientists or prospective hires, rather than rely on silly metrics like impact factor and # of reviews in "hot" journals. Why not try evaluating the person's science and complete contribution to the community? - Noah Gray

Ah, now I get you. How about, Dr Applicant lists "reviewer on eleventy-seven mss for Nature" on her resume -- would it also be OK for her to list someone like you as a contact to verify that statement, a kind of "reviewing referee"? If the committee contacted you, in confidence, would you then tell them what you know about Dr A's reviewing work? - Bill Hooker

I like where this is going. - Mr. Gunn

I would be most happy to have that discussion. It would be extraordinarily more informative for the committee than any quantification of review numbers. How long would that conversation be, 15-20 minutes? I mean it is essentially a "letter of recommendation" of sorts, right? I'm interested in whatever it takes to get the tenure/hiring decision handcuffed to real total scientific merit, which is not measured exclusively by C/N/S publications. - Noah Gray

It wouldn't necessarily need to be a discussion. Assuming, and this is a big assumption, that journal editors have the time and incentive to systematically assess every peer review of their peer reviewers, to come to some final “grade” of their work for that journal. Then they could just send those grades, or the average grades, to these committees that are asking for them, in confidence. Of course, the problem there is that each journal has different (minimum) acceptance standards. And each journal editor is somewhat different, and subjective, in their "grading". So it is still going to be highly subjective, while likely a more accurate approach than what Mr. Aspinall is originally suggesting. - Wobbler

It would help to have a publicly available database of reviewers shared by all journals, with some digital summary for each reviewer (number of reviews, average respond time, journal satisfaction grade). Journal names not to be disclosed for public, but available for the members of consortium of journal editors - genereg

I'd like to combine the best of the above suggestions, while simplifying a potential database and maintaining it for public consumption. The subjectivity pointed out by Wobbler is unavoidable, so I would propose a system that would not only signal value or pain regarding a reviewer's work, but also be of minimal impact upon the workload of the editors: RefDigg. - Noah Gray

Most reviews are solid and useful, but don't elicit either a euphoric or vitriolic response. But for those that manifest as one of the latter, the editor could "vote up" or "vote down" the performance of the reviewer. With all refs starting at zero, the reviews could distribute form there. Still subjunctive, but likely scored in relation to the other reviews on the same paper, so perhaps at least internally controlled (please don't skewer me on that one...). - Noah Gray

This would have to come out once a year so that individual ratings (or lack thereof) would be much harder to pin on a specific editor. The public would have a record of who is contributing to their field, and there would be less argument involved in deciphering more complicated rating systems. Basically, as a ref, you are rewarded or punished for being an outlier. - Noah Gray

However, in practice, I think journals would not want to contribute to such a platform, because they don't want to lose their competitiveness. Good reviewers are obviously highly valuable for journals. Why would high profile journals share this valuable knowledge with others? The journal peer review's primary objective is to help a journal decide whether work is worth publishing (after revision). That it is also a feedback system that helps authors/ peer reviewers improve their work is nice, but that's a secondary objective for journals, at best. Helping other journals? Yeah, right. - Wobbler

A clarification, the peer-to-peer post (which is my blog) Noah discusses here is itself a guest post responding to an Editorial and some published correspondence in Nature, therefore it is part of a continuing discussion. I would also like to mention in light of the discussion here that Nature's Editor in Chief personally wrote to a large number of our peer-reviewers to thank them for their reports during 2008, and encouraged them to use his letter as they wished (cvs, etc). I think it would be difficult to do as some are suggesting here and make a "public database of good reviewers" as it is all subjective and highly dependent on the particular ms and relevant knowledge of the particular refs chosen, and how appropriate they are. Nature actually has rather few repeat reviewers, we tend to spread the load very widely among the several thousand mss reviewed per year. - Maxine

@Wobbler Based on my conversations with investigators, new and old alike, they are inundated with reviewing requests on a regular basis, so it is not as if journals have any proprietary information with regards to reviewers. A simple PubMed search reveals the names of many appropriate reviewers. One editor's experience with a particular ref may differ from another, but I hardly see a digg-like loose rating system giving away secrets. - Noah Gray

@Maxine You're right, review assessments are subjective, which is why it might be easiest to rate only the extreme experiences (identifying distribution tails tends to be more accurate). An incentive for refs to provide the most useful review (and thus get rated) might enhance the overall process as well. There are lots of holes and flaws in such a system, so I am simply brainstorming potential designs. Reviewing is a lot of work and everyone should have a chance to be rewarded for it. - Noah Gray

A similar discussion took place some time ago at http://biology.plosjournals.org/perlser... on the basis of this paper: http://biology.plosjournals.org/perlser... . - Daniel Mietchen

Noah: That's good to hear :) If incentive is not an issue, then there remains the resources issue. I once asked a BMJ's editor whether they used their own peer review quality assessment tool systematically. He told me that, because it takes around 2 to 10 minutes per peer review, they simply do not have the time to rate/score each and every one of their peer reviews. So assuming that a decent assessment tool takes around 2-10 mins per peer review, will editors have the time to carry those assessments out systematically? - Wobbler

I don't get this 2-10 minutes. Seems a tiny amount of time to me. - Maxine

Well, as I recall in their case: up to 10 minutes a peer review, with 2 peer reviews per research paper. Roughly 1500 research papers being reviewed a year. That translates to 3000 * 6 minutes (let's say that's the average time needed for each assessment) = 300 extra hours. I guess this depends on the journal, but I can imagine even taking as little as an extra 5 minutes to review a peer review can amount to quite a bit of time in the long run. - Wobbler

The issue is not the time but what exactly we are talking about. My peer review assessment module is located b/t my ears and everytime I make a decision on a manuscript I am assessing the reviews. Therefore, I'm just doing my job, not requiring 300 additional hours... - Noah Gray

That's true, but I'm more referring to specific standardized "tools" such as checklists with scores. Things that will help keep those assessments more objective, or at least focusing on the same points. Example: http://jama.ama-assn.org/cgi... and more specifically: http://jama.ama-assn.org/cgi... - Wobbler

If you are interested in developing a workable web 2.0 peer-review process, please engage with the Sci-Mate project: http://www.sci-mate.org/wiki... - Christopher Dyer