If you seriously want to claim peer review is not reliable, then you should steer clear of the spurious "look at all the fraud that gets past" argument. Peer review cannot detect deliberate fraud and no serious advocate has ever said otherwise. Straw man, and a very annoying one. - Bill Hooker

Futher, extraordinary claims require extraordinary evidence. Given the intuitive value of peer review (as a first-pass validity filter ONLY), the onus is on the person claiming that it cannot do that job to provide evidence. By your own admission you have almost none. - Bill Hooker

Bill - in regard to your first point, I make a similar point in the post, although not as strongly as you. With that said, the fact that fraud does get past is not encouraging about reliability. As I say in the post, the line between honest mistake and fraud is sometimes thin. - Michael Nielsen

What's the extraordinary claim, Bill? Nowhere do I say it "can't do the job". My point of view is that it's a first-pass validity filter only. As I say in the post, many people believe that it's more significant than that. - Michael Nielsen

You state that "peer review is reliable" is a myth, call it a presumption and point to a lack of evidence for that presumption. I'm saying that the onus to provide evidence lies on the other side, the claim (which you are implicitly making, given your choice of phrasing) that it's NOT reliable. - Bill Hooker

You call the "can't catch fraud" argument weak, which it is not -- the thin line between deliberate fraud and honest error has nothing to do with the question of what peer review can or cannot do. Honest error is how science progresses. You provide one very weak further example, the BMJ one -- none of the data given has any use, the real question is, how many reviewers do you have to pick in order to catch all the errors? - Bill Hooker

Bill, until we're using the word "reliable" to mean the same thing, this argument won't go anywhere. As I try to make clear in my article, I'm using it to mean "catches errors nearly all the time, correctly validates most work, and doesn't suppress much innovation". You're using it to mean something much, much weaker; you're holding peer review to a much lower standard. I'll agree, incidentally, that if you use "reliable" in that weaker sense, then I wouldn't try to make this case... - Michael Nielsen

... but a lot of people use it in a much stronger sense than you do, and that's what I'm objecting to. - Michael Nielsen

If you are (and you are) claiming that peer review is not reliable, that is, "does not catch errors nearly all the time, does not correctly validate most work" (I'm avoiding the suppressing innovation claim, I think it's a larger and separate topic) -- then I'm saying that the onus is on you to provide evidence, which you have yet to do. - Bill Hooker

Wait, you're right -- "nearly all" and "most" -- OK, that's too strong a claim for what peer review can do, on any "ordinary language" understanding of the terms. Mea culpa, not reading carefully enough. - Bill Hooker

First, I don't think you can refuse to accept that peer review is working "reliably well" if it suppresses innovation. What if it suppressed 50% of all Nobel-Prizing winning work? Would you say it was working reliably then? Not that I'm claiming that it does, of course, just that I don't think you can sensibly talk about it working well if it's suppressing innovation. - Michael Nielsen

My earlier thoughts on peer review: http://cavlec.yarinareth.net/2004... and http://cavlec.yarinareth.net/2006... - D0r0th34

Bill - yeah, there's a _lot_ in the "nearly all" and "most", from my point of view. You probably won't be surprised to learn that I think your definition of reliable is the one that people _ought_ to have (well, except for the bit about suppressing innovation). Many do, but many don't, as well. - Michael Nielsen

I still say your post doesn't clearly put forward the "first pass validity filter, anything more is over reaching" point. It reads to me as an argument against peer review, period. Anyone else? This could be entirely my problem with reading comprehension here. (I'll stand by "fraud argument is strong, BMJ example is weak"). - Bill Hooker

I deliberately said I was avoiding the "suppresses innovation" topic, not arguing against it. :-) It's a perfectly valid criterion on which to assess peer review, I just don't have anything sensible to say about it. - Bill Hooker

peer review is *inherently* conservative, but that doesn't necessarily mean that it suppresses innovation - also, even though arxiv and ADS are very popular,most scientists still cite the published version - it's always been the case that you would know about research before the official jnl pub, but still rely on publication for certification, archiving, etc.... editors have always made the first cut and then sent only a selection to reviewers (see the Merton & Zuckerman work ~1972 using APS data) - Christina Pikas

In terms of further editing (and getting past reviewers ;-) I think Bill has a point about the rhetorical stance of the "reliability" section. Your section title implies that "peer review is reliable" is a myth therefore it is not reliable. I think your section makes a good case that the idea that 'the reliability of peer review is well established" is a myth but not the stronger statement perhaps - Cameron Neylon

wrt your last point, I have a great chart from my comps reading - i'll scan and post as soon as I can remember to get it in the same room with a scanner - Christina Pikas

@Dorothea: makes me v. happy to see someone else pimping Gigerenzer. Wonderful stuff. - Bill Hooker

Also think that the third section could possibly do with expansion - seems like there is much more to say about this - although perhaps the issue of communicating how science works to the general public is a separate and bigger issue? - Cameron Neylon

Bill - I updated the start of that section to make this a lot clearer. (The wording of the update may be a bit rough at the moment, suggestions for alteration welcome.) - Michael Nielsen

Christina - part of the reason for this is that many journals cut out arXiv citations, and replace them with journal citations. This irritates many people - see http://scienceblogs.com/pontiff... - Michael Nielsen

Nature Precedings got some items cited as well: http://network.nature.com/groups... - Bora Zivkovic

Cameron - A lot of stuff originally in section 3 got moved to my other essay on this topic, which is still to be posted. - Michael Nielsen

I like the new intro; I think what I'm looking for is a clearer statement of the position you take in this thread, that peer review should only be seen as a first-pass filter, and that assumptions that it can do more than that are ill-founded. (See also Dorothea's post on heuristics, linked above.) Still hating on the fraud argument, still want to know how many reviewers you need to pick to catch all the BMJ errors. Gotta work now, will check back later. Fun. - Bill Hooker

Bill - the intro has this sentence in it: "As an approximate filter that eliminates or improves many papers, peer review may indeed function well. " which I think pretty much sums up my position here. - Michael Nielsen

I am, by the way, delighted by self-referential nature of how much better my post has been made by post-publication review. :-) - Michael Nielsen

Bill, what do you want to know about the BMJ example? 221 reviewers found, on average, 2 of 8 errors deliberately entered into an article. - Michael Nielsen

Two things: what were the errors, and if you picked three of the reviewers (almost) at random (as a journal would), would you only catch two errors? I don't think so, since they likely didn't catch the same two. The full ms is available online at your link, and a quick scan indicates the authors were more interested in blind/not blind and signed/not signed -- they give no details of the errors and make no attempt to look at overlap in errors caught. - Bill Hooker

Also, 221 reviewers returned reports, 420 were invited. - Bill Hooker

Further: "Younger reviewers, those currently involved in biomedical research, those with more publications in the previous 5 years, and those who had received a higher grading for their past performance in writing reports by BMJ were all more likely to identify weaknesses in the manuscript." -- given that these are all factors that journals take into account when picking reviewers, my "(almost) at random" should be amended in favor of the actual process of choosing reviewers... - Bill Hooker

my sort of crappy paper on peer review from an sts stance: http://terpconnect.umd.edu/~cpikas... (hits the older lit) - Christina Pikas

Myths duly de-bunked, at least for the present audience (whatever that is). Maybe the next essay will have some suggestions for keeping the myths from cropping up. Journals/ed bds could do lots more to make people aware exactly what their review process is. Am thinking of a number of different discussions recently on library-oriented lists... - carolh

Recommend anyone interested in peer review read Christina's not-at-all-crappy paper as a good intro to a literature that is a lot larger than I suspected. (Plus, I esp liked the proposed improvements.) - Bill Hooker