Cameron Neylon
P ≠ NP and the future of peer review -
The online maths community has lit up with excitement as a document, claiming to prove one of the major outstanding theorems in maths has been circulated. In response an online peer review process has swung into action that is very similar to the kind of post-publication peer review that many of us have advocated. Is this a one off, a special case? Or does it point the way towards successfully using the web to find a way of doing peer review effectively and efficiently? - Cameron Neylon
Cameron - you mention the NaH discussions. The actual outcome from the blogosphere was much more interesting (and useful) than true or false. Instead of linking to these discussions the journal just retracted it without explanation, which isn't very useful for advancing knowledge. - Jean-Claude Bradley
JC, that's a good point. Not only was it quicker and more comprehensive, but also much more useful and informative. - Cameron Neylon from Android
OMG, *shivering*, theoretical computer science exam memories are coming back, can this be true? Honestly just reaching that proof level is already unbelievable, and if the gaps could be fixed, it would be just mindblowing. So, "Vinay Deolalikar you rock!", and @community "That might keep you, the smartest brains, busy for a while". - joergkurtwegner
Besides, please help voting for non-deletion on WP! - joergkurtwegner
I vote special case -- it's such an important problem that post-pub peer review works great because everyone is interested. - Bill Hooker
I certainly agree that it is a special case. I guess with some of these other examples I'm becoming hopeful that its not _so_ special that we won't start to see more of this going on - indeed there has been extremely little commentary that I've seen that seems remotely surprised by the process that seems to be being followed by the community. It's the fact that most seem to be assuming that this is a natural process that I find most interesting in some ways... - Cameron Neylon
Does anyone have a PDF copy, proof was removed from his web site? - joergkurtwegner
I don't think it's one of a kind, but I think that it is extraordinary or out of the ordinary. Probably <<10% of papers would warrant this kind of attention - Christina Pikas
If we agree that this is a special case, it seems to me we are seeing a lot more "special cases" than there used to be, making this kind of review a practical possibility, not just a theoretical idea. - Bonnie Swoger
One issue I haven't seen being given a lot of attention by bloggers is the issue of copyright. I'm assuming Vinay Deolalikar will have copyright over the TeX manuscript and PDF pre-prints, but maybe HP will have some sort of claim here? What happens if there's a bidding war amongst journals? Have scribd and the other hosting sites been issued with take down notices? Could the issue of copyright affect these collaborative peer review efforts if they quote sizable chunks or even the whole thing? What about the ethics of "leaking" the pre-print? (See also: - Dan Hagon
Further to the above, it's also possible that a handful of super-special special cases will whet the public appetite for this kind of process, so that we start to see it carried out for more and more "routine" sorts of discoveries. It would be nice even to get to the point where "extraordinary claims require extraordinary evidence" (say, the top 10 or 20% of odd/suprising/controversial/etc papers) brings to mind not just stringent peer review or submission to a Prestige Journal, but subjection of the claims in question to the scrutiny of the "peer crowd"... [edit: which was the whole point of Cameron's OP. Mea culpa, I'll try to remember to rtfp before I open my yap in future.) - Bill Hooker
Upon reading, "Most journals with commenting systems still get relatively few comments on the average paper" my immediate comment was, "The average paper is rarely worth reading, let alone reviewing." If people can't get at least several friendly colleagues to review their work, it tells us something about the scope and the quality, right there. You get to this point toward the end of the essay. Thank you for the story! - Maria Droujkova
every movement needs a figurehead, so why not take this example to figure well in Open Science - Claudia Koltzenburg
@ Maria: "If people can't get at least several friendly colleagues to review their work, it tells us something about the scope and the quality, right there." Not necessarily. Obviously, currently out of fashion research will attract less interest. But little interest doesn't necessarily imply lack of scope or quality - it could simply be ahead of its time. So, unfortunately, this reasoning only works one way. - Björn Brembs
+1 Claudia - Agreed, everything is in there, science, controversy, copyright issues. I would prefer a better life science example, but some maths+informatics example will do as well. Though, lets be honest the IP situations and relevance are totally different in those fields. So, we should never assume "one movement", its not that simple. - joergkurtwegner
@joergkurtwegner good point, it might still work in general, though, if the most general points (relevant for interdisciplinary perspectives, I mean) could be picked up for every example, btw there's quite a few useful bytes for this in - Claudia Koltzenburg
@Björn "it could simply be ahead of its time" yepp, fair point! - Claudia Koltzenburg
I'm not sure that it matters. One of the points of post-publication peer review is that it doesn't matter when it happens. Something might be harder to discover when it becomes relevant five years down the track than it would otherwise be but once it is discovered and reviewed that will improve. Much better than it not being published at all! - Cameron Neylon
hm, certainly at the stage of encouraging someone to get it out there, the "ahead of its time" factor is important, maybe crucial - imagine I wish to seek critical feedback from my most immediate peers who I trust for a first tentative round, what if I do not have them because of the "ahead of its time" factor? --- Once it IS up there: yes, I agree with your point @Cameron - Claudia Koltzenburg
@the-"ahead of its time"-factor: maybe like the inventors of double-blind peer review we might have to discuss anonymous authorship for Open Science post-publication review scenarios - anyone seen ideas & discussions on this one? - Claudia Koltzenburg
wondering if you guys have seen <a href="">this article</a> about the european research effort "liquidpubication"? features the sentiments, "don't print it, post it" and "don't review it, use it". the latter especially is a really interesting point, although perhaps easier in computer science than, say, high energy physics research-- i'm not going to just go and build my own LHC... but, "using" it could also mean referencing it in another paper that builds on those results. of course, you don't want to build on them if they are wrong... - jessykate
perhaps a form of "using it" could also be in showing it's wrong. that's interesting because it gets at the question of incentive, and also of quality. if i review a paper and reject it (especially in an anonymous peer review system), what professional credit (incentive) do i get? especially in an open peer review system where i probably don't even get my name on a list of editors. but if i write a (micro?)paper showing it is right or wrong, i keep the emphasis on implementation (which is arguably the point ultimately), i create link back to the previous paper's efforts (right or wrong), which creates a record for science, an acknowledgement (in the peer review "graph") of their work, and some incentive/recognition for me, too. - jessykate
@jessykate, re @liquidpub see - Claudia Koltzenburg
very interesting point about ''using it" @jessykate - let's look at what disciplines may share or maybe not: what is a use case in a given field, and what is the "it", respectively? - Claudia Koltzenburg
Claudia - agree this will be a difficulty, but if someone really can't get feedback from _anyone_ on work then I think they've got bigger problems than being ahead of their time. No-one works alone any more - if nothing else you should be able to get local colleagues to take a look. @jessykate My view is that using it is inherently a form of review, as is citation, or any other reference, if we could capture them. The LiquidPub people are certainly pushing this angle and I agree with them philosophically. I'm less sure about their technical approach but I have to admit I haven't looked in detail for a while at what they are doing. - Cameron Neylon
Cameron, partly agree, which disciplines do you have in mind? - Claudia Koltzenburg
@Claudia - Fair enough, thinking mainly of experimental sciences. This is clearly less true of maths and theoretical physics, where people do seem to largely work alone. But there there seems to be a more positive culture of review both prior to submitting pre-prints and around the pre-prints before formal publication. And of course it remains the case that the lone non-professional genius is out in the cold in these situations unless they can find a way in. Somehow we need to protect those researchers whose time is most valuable so that they only review things that are a good use of their time. The problem is that they may be the only people who can tell that. I think at the end of the day that anyone who is expecting someone else to contribute their time to reviewing their work needs to demonstrate willingness to do the spade work to get some credibility in the community but we need to provide good mechanisms so they can then build on that credibility by getting good quality reviews of their work. - Cameron Neylon
The post was picked up on Techdirt: Some comments there interesting although much of it similar to what we've seen before. - Cameron Neylon