Sign in or Join FriendFeed
FriendFeed is the easiest way to share online. Learn more »
Shirley Wu
Overheard regarding papers published in PLoS ONE - "it was rejected somewhere else", "The bar is 'not crackpot'", "people publish in CNS because that's where the attention is, I don't know anyone who reads PLoS ONE", "The reputation of the journal is a good way to filter out noise". Is there truth to these claims? Discuss.
Almost all papers, in all journals, have been rejected from somewhere else. Our bar is "is it science, is it conducted properly, is it reported properly, and do the conclusions follow the data etc" - the bar is not "is it sexy, or impactful, or a major advance". At the same time, we are not CNS - as we are not selective. CNS combined publish just 5,000 articles a year between them but there are over 1.5 million articles published per year - the universe is not just CNS. The reputation of the journal is ONE way to filter out noise. But is it the best way? (it is well reported that 89% of Nature's Impact Factor comes from 25% of their articles - so was Nature inclusion a good filter for the other 75%?). We think we have new alternatives (http://everyone.plos.org/2009...). In a couple of weeks we will be putting our usage data online for anyone to see exactly how many people read our articles. Will CNS be doing the same? - Peter Binfield
Peter, I certainly don't disagree with what you're saying and think PLoS ONE is valuable and innovative. But I was wondering if these negative judgments are pervasive (FF/twitter is a bit of an echo chamber and the real world can be a shock sometimes) and if so how to change them. There are those who argue that CNS has high precision even if they miss some good papers and so it's more worth their time whereas PLoS ONE might be sensitive (because "they publish everything" - or so they think) but their false positive rate will also be high. The bit about Nature's impact factor coming from just 25% of their articles is good to point out for those folks. - Shirley Wu
The problem with echo chambers is that the Internet echoes forever; and forever is a long time. We just need to push out as much positive info as possible to try to combat any negative comments which may have been made rashly, or in error, but which get re-referenced for eternity. Our article-level metrics program will presumably show people whether any given paper in PLoS ONE is 'high' or 'low' quality and so they wont have to rely on the filtering ability of a journal to tell them whether it is good or not. The ref to that Nature quote is: http://www.int-res.com/abstrac... - Peter Binfield
There's this thing known as FUD. Happens when someone sees their status eroding. The whole PLoS articles are not as good is just that, FUD - Deepak Singh
Indeed, it may be fear, uncertainty, doubt. It may also be lack of information and hard data. We are going to fix the latter. Certainly, people are voting with their feet - we have 37,000 published authors in under 3 years, and people are publishing with us in ever increasing numbers (http://poeticeconomics.blogspot.com/2009... ) - Peter Binfield
"it was rejected somewhere else" - perhaps. This is hard to tease out, but I have a feeling that most of the manuscripts that come to PLoS ONE have never been submitted elsewhere - Bora Zivkovic
"The bar is 'not crackpot' - good bar, IMHO. Why is any other bar necessary? Think. Really. - Bora Zivkovic
"people publish in CNS because that's where the attention is, I don't know anyone who reads PLoS ONE" - who still reads journals? Srsly? Don't people search online for papers they are interested in? Do physicists read biology papers when their copy of Nature arrives? No, they read Nature for "news and views". - Bora Zivkovic
"The reputation of the journal is a good way to filter out noise" - perhaps a century ago when every scientist could read every scientific paper and understand it, and every scientist was a 'Victorian scholar' who felt the need to keep up with ALL of science. Today, you read papers in your narrow field - you find them online. News from other sciences you can find in pop-sci magazines, on blogs, etc. - Bora Zivkovic
@Bora, I think there are still a fair number of people who don't search for papers necessarily, but browse TOCs, and so only browse the journals they're familiar with. During the discussion, someone asked, baffled, "but there are already so many papers [without PLoS ONE publishing so many more], how would people find ones of interest to them??" - Shirley Wu
A related discussion - based on a correspondence in Nature by a proponent of the views Shirley cites - is at http://ff.im/4GWlM . - Daniel Mietchen
@Shirley - although ToCs are certainly an important discovery tool, any publisher will tell you that the vast majority of their usage comes in from Google (who then read an article, and leave again to run another search) - Peter Binfield
@Peter, that would make sense, but I'm wondering if that necessarily translates into Google being the majority of people's preferred method for finding papers. At least the impression I got from folks in my lab was "so many papers, so little time" and so they're skeptical of anything that adds to the glut of papers without clearly adding value. They might agree on the principle that scientifically sound results should get published, but they'll disagree on whether those papers are important enough for them personally to spend time sifting through them. - Shirley Wu
They also think, "if [a peer reviewer] didn't make a value judgment on whether this paper is significant, why should I waste time reading it?" - Shirley Wu
"I'm wondering if that necessarily translates into Google being the majority of people's preferred method for finding papers" - Good point. I guess you would want to measure time spent on page by people who come via the 2 (or more) routes to see how targeted their interest was - Peter Binfield
@Shirley - Then they are admitting that they would prefer one (or perhaps 2 or 3) other people to decide what is important for them, and so decide on their behalf what they should be reading. Doesnt sound like a very informed way to filter imho... - Peter Binfield
@Peter, more that a million people access papers through google 25% of the time will mean that publishers see google as a huge source of traffic, but doesn't mean that people think of google as their preferred method to find _NEW_ papers. - Shirley Wu
@Peter, well, it's using expert opinion. We all use it to some extent in areas we're not familiar with. If people aren't that internet savvy or aren't that organized, they depend on other people or name-brand journals to bring things to their attention. Also, commenting on papers hasn't really taken off yet - just a matter of time, probably - but it just means that the post-peer review process hasn't really proven its value yet. - Shirley Wu
"but it just means that the post-peer review process hasn't really proven its value yet." - indeed, and we DONT view our efforts as post-pub peer review. We view it as a new way to do post-pub evaluation / filtering / discovery. - Peter Binfield
Oh, the other thing that someone mentioned was "comments are valuable" - meaning "why would I give away my intellectual capital?" People are willing to share their comments with their labs or close colleagues, but not to the public or to the general scientific community. Is this just another mindset we combat with positivity and action? How to combat the vicious cycle of "no comments, so no value", "no value, so i won't comment"? - Shirley Wu
"Is this just another mindset we combat with positivity and action?" - I would say we combat it by showing them the power of being open about these things. For example, social bookmarking only works when everyone shares their bookmarks - in this example there is a clear benefit to both contribute and use. If people realised that by leaving comments they would be advancing science; preventing people from repeating mistakes; building a personal reputation; accelerating discovery etc etc then they would start to do it more (I presume). - Peter Binfield
@Daniel, ah yes, I remember that thread now. Unfortunately I think many scientists are similar in mindset to the letter writer. They don't know about or understand new ways of receiving content, which might seem strange to those of us here, but there are many more people out there than are in here. - Shirley Wu
Also, you could use the same argument about peer review: "My time and thoughts are valuable, why should I do peer review". Apparently academia feel that the quid pro quo works in that situation at least (and that is done anonymously!) - Peter Binfield
@Peter, true, though I think some of that is tied to the reputation of the journal again - being a reviewer for Nature > reviewer for PLoS ONE (in their eyes), editors know them, they can talk about it and gain status. They get tangible and subtle career boosts. Whereas commenting on papers online and publishing in PLoS ONE doesn't get someone tenure (yet). "It would be very brave and courageous for a junior scientist to do these things, but they'd need to depend on luck getting them a hiring committee that thinks the same way." - Shirley Wu
Peter - given the problems that journals have finding suitable reviewers, I would hesitate a bit calling that a working system. - Daniel Mietchen
@Daniel :) - Peter Binfield
Another link that may be useful reposting here: Pubfeed at http://pubfeed.cs.toronto.edu/ basically allows you to treat the whole web of scholarly articles like a TOC alert (just a bit more customizable) and pipe that into your preferred feed reader. - Daniel Mietchen
@Shirley - please dont forget that there are 25,000 journals in the world and millions of papers published per year. CNS is just 3 titles, and if you lump together all similar titles (highly exclusive, professional editors, well known brands, conferring 'bragging rights' on anyone who works with them) then you are still talking about just a handful of the titles, with a small percentage of the content. We need a system that works for everyone, not just a small sub-set - Peter Binfield
@Peter, oh, I'm well aware, just parlaying bits of an impromptu debate I had earlier today with people who don't see the value of venues like PLoS ONE. These are all arguments they make, and while I don't agree with them, it is tough to convince people - Shirley Wu
You could try asking them exactly how many downloads their last paper in a 'high impact' journal got... - Peter Binfield
Fair enough, but you know, I really don't think they think about that. They think "what will be in my CV?" and they think any journal that is somewhat competitive [includes other PLoS journals, BMC journals, etc] looks better than one that accepts anything that's methodologically sound. Again, not my view, but perhaps one that is held by many. Do people list # of downloads on their CV for publications? - Shirley Wu
They dont, because they dont have the data. However, people do list if their paper was rated by F1000; or if BMC designated it a 'highly accessed' article. So I think they will start to say "this paper was downloaded 5000 times in the first 3 months which put it in the top x% of all PLoS ONE articles, the top y% of all PLoS articles, and the top z% of ALL articles" (when the rest of the world starts quoting this data) - Peter Binfield
I think of it as trying to set up a social experiment. If I'm right and a more cooperative model can produce better science than the current hypercompetitive structure, then over the next decade or so, facility with new methods and metrics that center on Open practices will provide a competitive edge for some researchers, and unwillingness to change will put others at a disadvantage. We don't have to convince everyone -- in fact it's bad for the experiment if we do, no control group. We just have to convince enough people to test the Open Foo waters so that we can distinguish between "competition turned out to work better" and "it just never got off the ground". - Bill Hooker
And isn't that the scientific way? - Deepak Singh from IM
"I think there are still a fair number of people who don't search for papers necessarily, but browse TOCs" Could it be that those are the people publishing in CNS and miss the most important papers for their work? http://bjoern.brembs.net/comment... It's only one example, as anecdotal as it gets, but it shows two things: 1. CNS 'quality' is merely correlational and highly noisy. 2. Missing just one important paper can break your publication. @Shirley: I think you need to show your colleagues how to efficiently search for literature... - Björn Brembs
My environment is rather positive about PLoS ONE. We all know about the difference between relevance and quality. While many PLoS ONE papers might not be of widespread interest, the review process is of comparable quality or better to smaller conferences and e.g. high volume BMC journals. Other journals have severe issues with reviewer quality these days, and it seems to become worse. - Roland Krause
I'm still wondering about the degree of scalability of post-publication (significance) peer review systems. Is it really realistic to think that once (all) journals go OA and implement such a system that the entire scientific community will benefit? Assuming that it's "fair" for all journals to get equal amount of attention from "scholarly feedback communities", how can we encourage (and even enforce) that to happen? Since PLoS ONE is not specifically publishing papers of perceived high significance/originality, how do we make sure other (or all) journals employing the exact same system get a fair shake? When it comes to helping readers find relevant/significant papers, I don't see how this system is going to be an improvement over the traditional system. It may be just as difficult, if not more difficult, for qualitative journals to stand out when they don't have an established feedback community to get those quality indicators that will attract scholars from viewing the articles in the first place. And I haven’t even talked about enforcing or even verifying the accountability/ answerability of such feedback. - Wobbler
I agree with Bill Hooker's statement just above. Those who echo Shirley's original quote will be at a disadvantage, which means better odds for the Shirley's of the research world. - Jason Hoyt
I often say something along the lines of what Bill said. The environment is changing. To succeed in the new environment, one has to change not just one's publishing habits, but also rethink how to do research and how to write it. Thus, people who think about it early on will be able to gain advantage over people who are still stuck in the old ways of doing things. As the new environment forces hiring/promotions committees to change their criteria, and thus force researchers to change the way they write their CVs, the folks who did not change their research/writing/publishing strategies early enough will be at a disadvantage - at least at a disadvantage in comparison to what they thought, by old rules, where they stood in the hierarchy. - Bora Zivkovic