Glyn Moody always finds a nice way of pulling the story together and making it clear what is going on. Kicking myself that I didn't connect this to the scarcity argument so explicitly... - Cameron Neylon

I would argue that the Internet is good at peer reviewing the *impact*... but I would not immediately see it as replacement for peer-review of the presentation; just thinking at the latter process, peer-review of source code might be an example of what we can then expect... nothing much. I would agree that for peer reviewing the impact there is abundance, but for peer-review the details, the market is still very, very, very scarce; the market of people qualified to do a proper in-depth peer review of some scientific output is just small; there might be more scientists, but there are many more specialistic research fields. This why I keep stressing that we need to make machine readable publications, to weed out most of the common mistakes, before a human starts looking at it. - Egon Willighagen

You'll get no disagreement from me on the latter, but it seems to me that for any argument that says there are few reviewers competent in specific areas, there is a benefit in expanding the range of people available to become competent. I think it is pretty rare that any pre-publication peer review effectively assembles the ideal range of expertise and competence - so better to allow downstream commentary at least. - Cameron Neylon

What makes you think that 'the scientist' will start reviewing papers outside his field, or just have the time for that, or is willing to overcome bad English from non-native English authors? Where is the incentive? The thing publishers do at least, is introduce this much-talked-about game theory aspect: it pays off to review papers (to see more new research than others; to build up fame; ...). Put differently, where is the 'reviewer level impact' at efforts like PLoS ONE? - Egon Willighagen

"it pays off to review papers" -- I disagree. I never got to see much new research that was close enough to my field to be interesting, certainly not enough to compensate me for the time I spent, and there is NO reputation system for reviewing. It is completely unrewarded and unregarded. Ask the question in reverse: how would a scientist suffer who rejected all requests to do peer review? Answer: she wouldn't, she would prosper by having so much more time to work on her own stuff! - Bill Hooker

I'm pretty much with Bill on this. My suspicion is that if you turned around today and asked people to suddenly do peer review they'd laugh at you, say they were too busy. We do it because we've "always" done it - not I think out of any altruistic motivation, nor particularly out of a selfish one, but because we're worried about being caught out in the situation Bill describes - having rejected all requests. Now being an _editor_ is a bit different - that looks good on a CV, but I certainly don't put down the journals I've refereed for on my CV. - Cameron Neylon

@Bill: "It is completely unrewarded and unregarded." Do you see this abundance then, that Glyn is talking about? - Egon Willighagen

@Egon: If I understand Glyn, he is saying that (1)peer review is only good as a first-pass filter, and (2)that level of filtration is no longer needed, since publishing is cheap and easy. A coarse first-pass filter made sense when publishing a crap paper meant wasting scarce resources; now that it means wasting only a few electrons, it makes sense to position the filter on the other end: between the mass of published work and the user who wants something out of that mass. So the abundance he is talking about is not in peer review, but in publishing. He is saying, I think, that there is no abundance of peer reviewers; rather, that is a very scarce market, so why not bypass it altogether, because there is an abundance on the other side: publishing resources, and post-publication filtering. - Bill Hooker

@Egon, cont'd: I agree with (1), as I paraphrased Glyn, but not with (2). I do not think as highly of post-publication filtering as Glyn seems to -- for instance, as you point out, PLoS ONE's article level metrics look like a good way to review impact, but do not provide feedback on details. My solution would be to *change*, not bypass, peer review: (1)make it a recognized and rewarded part of the job, and (2)only ask peer reviewers to look at the science, not make guesses about the impact which is better left to the post-publication filters. - Bill Hooker

It's the scarcity of publishing that's the core issue that Glyn is referring to. But longer term I just can't see pre-publication review being either affordable or useful. There is very little evidence that it does any good at all - and even if it does the same task would surely be better done by more people over time. The question tho as Bill points out remains one of incentives and motivation - whether before or after publication the key issue is how to motivate people to review and how to capture as much value as possible out of that review process. - Cameron Neylon

The first thing that has to go is the idea that a reviewer can predict the "impact", importance, of a paper. After that I think, with the proper incentives, peer review could do a lot of good. As soon as you can use your peer review record as part of your argument for tenure or promotion, I suspect the quality of review will dramatically increase. That record could be a number of things -- no. of mss reviewed, letters of thanks from journals, no. of times your posted reviews got a "useful" vote, etc etc. - Bill Hooker

Think of it this way: PLoS ONE aims to publish anything that IS publishable, and they still have a 30% reject rate. That's a LOT of crap that post-publication filters never have to deal with. If we spare reviewers the guesswork about impact and make quality reviewing a visible and valuable part of being a scientist, I think there is still a lot of value in that coarse, first-pass filter. - Bill Hooker

Or, turn it around: if we take away that first hurdle of peer review and publish anything, will there not be an absolute flood of crap? Think about some of the papers you've reviewed -- cut and pasted from half a dozen reviews with two poorly photographed gels masquerading as data. OK, it doesn't take long to spot that as crap -- once you've downloaded it, that is. You can't tell from the title. And then think of other papers that were complete bollocks once you got into them, but it took an hour or so to be sure just because the arguments were nearly right and the shoddy data looked good at a glance, and so on. Even if peer review saves readers that much time, I'd say it's worthwhile. - Bill Hooker

(I don't like how Scholarly Kitchen that last comment of mine sounds, with the "flood of crap" and so on, but it's high time I shut up in this thread so I'll just leave it for others to tear at.) - Bill Hooker

@Bill, just on the issue of peer review for assessing impact it's worth listening to John Wilbanks' BL talk from last year about this (at about 13mins http://www.bl.uk/whatson...). - Dan Hagon

@Dan, thanks. Briefly: peer review aimed at assessing impact is subjective and therefore favours the dominant paradigm. - Bill Hooker

At the danger of sounding like an old grammophone (cf. http://www.plosbiology.org/annotat... ): Wouldn't making the review process public (which does not exclude reviewers remaining anonymous if they wish) solve much of the "it is completely unrewarded and unregarded" problem? Then public lists of reviews could be generated (possibly automatically), and reputation could grow from that just as it does from papers and database entries. Some publishers already go in this direction (e.g. http://www.egu.eu/ ). The tricky part, in my view, is to get funders and tenure committees to recognize such non-"paper" contributions to the advancement of science. - Daniel Mietchen

@Bill, "there is a lot of value in the course, first-pass filter"... but I just don't know if it is a good use of time. Perhaps I'm wrong because I'm new at this and haven't seen a whole lot of rejection reviews, but writing rejection reviews still takes time... way more than just realizing something is junk. Must compose reasons for rejecting, constructive criticism. Not to mention administration of reviewers. If something is crap, it might save the scientific community net time if that were determined by the first ten people who read the paper all quickly give it 1/5 stars, or fail to bookmark it, or whathaveyou. We all end up on crappy websites every day... glad there is no crap filter there.... - Heather Piwowar

I'm with Bill and Daniel here. Incentives and reputation are the key! My personal experience with peer-review is twofold: one, there are few papers I read so thoroughly as the ones I have to review. Two, my papers usually improve by forcing someone to actually read it thoroughly. So an incentive to more than just casually read a larger section of the literature than you normally would is definitely good for science, even if the benefit may lie outside the parameters measured in the cited studies, or be too small or subjective to measure. - Björn Brembs

National differences? In NZ we are expected to put the journals for which we provide peer review. It counts as 'peer esteem' for annual performance reviews (and I think also for the ranking system that determines govt funding to each university). - Kubke

I really disagree with the post. We need peer-reviewing and the current post-publication peer-reviewing is not working well enough to do away with the pre-publication quality controls. From my experience, my own papers have been improved by peer review and I certainly read papers I am reviewing much more critically than any other paper. Again . .we have discussed this before ... I can read critically something that is close to my area but we all must acknowledge that we need help reading articles that are away from our core expertize. On average I am convinced that peer review adds value. Even the subjective evaluation of editors and reviewers on impact adds value. If there was only pubmed with no sub-sections (journals) I would not read anything else but those things triggered by my favorite keywords. Post-publication evaluations hold the promise to do away will all of this but it is not working yet. I get some value from doing peer-review since I am asked to review on areas that I am interested in but I also see it as part of my duty as a scientist. - Pedro Beltrao