 |
|
| FriendFeed is the easiest way to share online. Learn more » |
Jan Aerts › Likes
IMO this is possibly the single most useful thing we could do to enhance the current literature: BMC Blog "Towards threaded publications" - http://blogs.openaccesscentral.com/blogs...
Steve Koch, Garret McMahon, Just Joe and 13 other people liked this
If we could link forward into later developments related to any given paper and place it in context it would solve many of the current system. If one of the major criticisms of C/N/S is that they publish "interesting but wrong" papers then this would be a strong motivation to authors to make more effort to ensure they're right. And at the other end it will really help to highlight papers that weren't high prestige when published but turn out to be high value further down the track.
- Cameron Neylon
from Bookmarklet
See also: http://cameronneylon.net/blog...
- Cameron Neylon
Indeed!
- Kubke
I like this approach better than some of the proposals I've seen for so-called "living" journals where old content is replaced by new corrected content. At least with this approach there is an historical trail.
- Jenny Reiswig
Isn't this what we've been arguing for a long time now: why aren't the citations listed on every paper just as the references are listed?
- Björn Brembs
What I'd love to see along with this is a versioning system. This preserves the all-important "historical trail" Jenny mentions, but also allows for more complex changes. How cool would it be to fork articles, a la Github. If you didn't like the direction an article was going, you fork it and, with the underlying data available, revise and update it yourself. Later it could be potentially merged back into the trunk.
- Jason Priem
Jenny - such living journals already exist. Google "Living Reviews". Living Reviews in Relativity is quite influential, and has been going for a long time.
- Michael Nielsen
Here's something fun - http://relativity.livingreviews.org/Article... The latest (2008) version of Carlo Rovelli's 1998 article on loop quantum gravity. The citation tab contains some useful discussion of how to cite different versions. (The physics arXiv also has this problem.)
- Michael Nielsen
Living Reviews is CC BY-NC-ND on the bottom of that page, or © Max-Planck-Gesellschaft according to http://relativity.livingreviews.org/About... (linked from the history).
- Daniel Mietchen
following from "fun with citation" session at #scio11 yesterday: both forward and backward citations would be much improved with context, i.e. what part of forward or backward citation is relevant and why.
- Steve Koch
I need version control for the wiring in my house.
Florian Bahr, Jan Aerts, Cameron Neylon and 2 other people liked this
Clojure Cookbook - Getting Clojure - http://www.gettingclojure.com/cookboo...
Rajarshi Guha and Jan Aerts liked this
Welcome to the Clojure Cookbook. We have collected a number of recipes which show how to accomplish specific tasks using the Clojure programming language. This site is intended to be a community resource, and your participation is vital to making it work. If you have examples that you would like to contribute please do so. Likewise, if you have a better way of doing something illustrated here show us your approach so that others may learn from your experience.
- Pierre Lindenbaum
#openrescomp promotional slide
Yaroslav Nikolaev, Jan Aerts, Graham Steel and 2 other people liked this
Open Research Computation: An ordinary journal with extraordinary aims. - http://cameronneylon.net/blog...
Mr. Gunn, Greg Tyrelle, Heather Piwowar and 5 other people liked this
I spend a lot of my time arguing that many of the problems in the research community are caused by journals. We have too many, they are an ineffective means of communicating the important bits of research, and as a filter they are inefficient and misleading. Today I am very happy to be publicly launching the call for papers for a new journal. How do I reconcile these two statements?
- Cameron Neylon
playing with git-tower beta (OSX).. it's very pretty: http://www.git-tower.com/ - GUI+Git is still a bit a misnomer (for me) though.
Jan Aerts liked this
Grant Writing Boot Camp - http://bytesizebio.net/index...
Jan Aerts, Khader Shameer and Mickey Kosloff liked this
How to Switch From SVN to GitHub for Small Teams - http://www.readwriteweb.com/hack...
Piotr Byzia, Jan Aerts, Pierre Lindenbaum and Cameron Neylon liked this
Ok crowd-sourced proposal time. Topic: Research Metrics. Aim: Develop and deploy credible metrics frameworks that encourage "open" behaviours. Value: $30k. Go.
Barbara J McDonald, Carl Boettiger, Peter Miller and 24 other people liked this
More precisely: I have an agenda, some ideas, and a possible line on about this amount of money. Am interested in any ideas on how most effectively to create change with about this resource and anyone who is interested in taking part (with the caveat that this is not a whole lot of money if it gets split a lot of ways).
- Cameron Neylon
Particularly interested in bringing existing projects/initiatives together to maximise effect. I'm aware of a range of things going at the moment but would appreciate pointers to any others.
- Cameron Neylon
Quite. More downstream funding might be possible if there was demonstrated value out of the "pilot" or whatever it might be. There are a bunch of JISC funded projects that are sort of, but not really, in this area at the moment (they are focussed on citation) so that's another place to start.
- Cameron Neylon
Do we have any existing examples of metrics that encourage open behaviors?
- Mr. Gunn
Well I would argue that re-use metrics (downloads, data citation) push in the right direction. I couldn't point to hard evidence to support that tho.
- Cameron Neylon
Perhaps the pilot could be to gather such evidence?
- Mr. Gunn
Thomson-Reuters shares went down hard in the last 26 minutes.
- pn
@D Peter Binfield has talked about Google Analytics style tools provided by a third party to do cross publisher ALMs. Something I always thought was an interesting idea. It could at least provide a consistent way of dealing with that kind of gaming?
- Cameron Neylon
@Mr Gunn: It would be difficult to do that prospectively in the time frame of a small grant. Can you think of clean examples which could be looked retrospectively? Key issue is actually persuading funders to adopt something of course.
- Cameron Neylon
@Paulo: Don't worry, my next idea will take out Pepsi...
- Cameron Neylon
No idea how you'd do it but what I'd really like is to understand _why_ someone downloaded something. Was it for the figures, a specific piece of data, or for the beauty of the whole work...
- Cameron Neylon
It's that or some sort of single use citation token as Claudia (I think) suggested at one point. Either would likely lead to bias anyway...
- Cameron Neylon
Dorothea - Have you looked at the PIRUS2 and MESUR projects? They are looking measuring usage statistics from mulitple locations (PIRUS) and assesing quality of content (MESUR). They already have done a lot of preliminary work. I would suggest usage metrics as they both argue usage happens earlier in the research cycle than citations. Is there a way to look at any open behavior at the grant application stage, like this thread? That's even earlier in the research process.
- Elizabeth Brown
Cameron, if I could think of a way to do something like that retrospectively, I'd already be doing it ;-) Agreed that usage metrics are the way to go, though, to move upstream from citations.
- Mr. Gunn
Interested that no-one yet has explicitly brought up the issue of data that metrics are based on...
- Cameron Neylon
@Mr Gunn, yes but how far upstream do you think it is possible/sensible to go?
- Cameron Neylon
Hate to say it, but I think your funding is short a few zeroes. If this were something you could do for $30K, PLoS or BMC or even Thomson Reuters would already have done it. But so as not to be all negative all the time: how about expanding Heather Piwowar's work on data re-use? The aim would be to compare the rewards (citations, collaborations, etc) reaped by open, re-usable datasets...
more...
- Bill Hooker
It's alright. I'm not proposing to tie up all the loose ends for this kind of sum. Although I would say that in many cases PLoS/BMC don't have that kind of spare cash just sitting about to do something different. Question to my mind is how to make the most difference with that kind of amount...not necessarily to solve the problem.
- Cameron Neylon
from twhirl
The question with this kind of thing is always how best to take a small amount and multiply it with existing funds that other people already have...
- Cameron Neylon
In that light, I think my dataset idea looks a little better: you might be able to tie in to what Heather is doing now. The real trick would be to get a few *more* related efforts coming together under the same roof.
- Bill Hooker
Certainly I'd be pretty happy with building a reference dataset of data citations and/or uses.
- Cameron Neylon
I agree that a) this is a very important issue, and b) $30k is not a lot of money. I would use the money to talk to the stakeholders (researchers, institutions, funders, publishers, vendors) - and that would include Thomson Reuters - and write a report that summarizes the key issues and comes up with a set of recommendations. On a very small scale I have started to do just that (Cameron knows about this).
- Martin Fenner
Agree with the talk to stakeholders. Less sure about the writing of reports...but that's maybe a personal bias...as a focus for discussion yes, but as an end in itself? But definitely agree that engaging the relevant people is critical. There is no point building/writing about anything that doesn't serve the needs of those whose metrics will be taken seriously by researchers...
- Cameron Neylon
Reports indeed have a bad track record. Maybe principles and action points?
- Martin Fenner
A set of agreed principles would be a very interesting outcome if it were feasible. Action points seems easier but then actually turning them into action becomes the challenge...
- Cameron Neylon
About citations of data and other open science outcomes like code, can current citation metrics make a separate count for numbers of citations of data and code as opposed to articles? Is there some agreed meta-data field that indicates something is data rather than an article? If someone wants to reward open science, then we need to be able to count and aggregate open-sciency citations.
- Alex Holcombe
Yes, progress being made on citing data but citing data within a publication rather than the publication itself is some way off. And citing code is still pretty rare (partly because "publishing" code is still pretty rare). Martin and Mummi did a nice double act at Science Online London looking at the use of ORCID to mix and match data and paper citations.
- Cameron Neylon
I would suggest to talk the David Shotton (Oxford) on CiTO, and get citation types into the equation... and develop a simply H-index-like thing that is weighted for citation types... type: usesMethodIn -> large positive for H'-index... if type: refutes -> negative (or really small positive) on H'-index...
- Egon Willighagen
To my mind the best thing would be a pilot where you pay for (part of) a person or student to piggyback on an established database with appropriate data sets (at EBI, for example, this would be something like PRIDE or ArrayExpress) and track downloads then provide example chunks of end-of-grant-style reports of impact and write a paper or three on it (one on mechanics, one with submitters, one forward-look), etc.
- Chris
from twhirl
interesting. thinking. fwiw, I have a research proposal drafted and submitted, proposing to lay groundwork on patterns of data citation and reuse tracking. Might could be synergistic? Submitted for what is supposed to be a double-blind review process, so in an attempt to make it ungoogleable I'll just put a link here: https://docs.google.com/documen...
- Heather Piwowar
I also have a large set (11k) of "data creation" papers whose citations we could follow, examining for data reuse patterns and citation benefit of sharing. Totally agree on CiTO. Ideally would supplement with data creation papers in other fields/datatypes too, to illustrate differences across fields and datatypes. Not easy, but have some methods. Also, could work with GEO and...
more...
- Heather Piwowar
Made me think of something that might be complementary: to interview a set of data producers who have submitted somewhere about their attempts to make their data traceable, to claim credit in subsequent funding proposals, and how they think that played out. And ideally, to then revisit those applications with the funders to whom they were submitted.
- Chris
from Android
I think anyone submitting a proposal like those above should look at the MESUR project in particular- they've already looked at usage stats and compared this to article citations to estimate the quality of work. Some of the questions above have already been answered. My question is, can a list of open behaviors be clearly identified? I think that might be more challenging than the metrics, since most of the tools are evolving so quickly.
- Elizabeth Brown
MESUR definitely high on my list (as is PIRUS now) and will also talk to Carl Bergstrom as well. Also liking Chris and Heather's comments.
- Cameron Neylon
from twhirl
I would like to extend something like http://wikidashboard.appspot.com/ (plus http://orcid.org/ - once workable) to contexts like http://www.science3point0.com/coasped... . Might solve the on-wiki part of the problem.
- Daniel Mietchen
Are you crowd-sourcing tips/pointers/suggestions for your own proposal or are you asking people to contribute directly to a 'community' proposal (via google docs for example), with the intention that people who contribute to the grant get a slice of the funding ?
- Greg Tyrelle
Greg, A bit of both. I have my own ideas and want to make sure I haven't missed anything obvious but also am looking to try and form a network of people to take this forward in some form. I'll move onto an online document in the next few days to which people are welcome to contribute with the idea of taking part. Bottom line is that at this level there won't be money for much beyond maybe travel/meetings and one FTE for a couple of months.
- Cameron Neylon
My basic idea is meeting with stakeholders followed by hacksession "sandpit" out of which the group decides to fund someone already in post to be extended to take most promising aspects a bit further. Open minded as to whether that aspect might be liberated data, some sort of mashup, or a protocol/framework, or a document.
- Cameron Neylon
Would love to participate. Sounds interesting.
- 'Mummi' Thorisson
Throwing my $0.02 worth into the pool: how about considering also attribution and metrics for less granular contributions than papers/datasets: e.g. output of curator activities, or 'nano-publications' mined from the literature? (http://www.nbic.nl/uploads... http://laikaspoetnik.wordpress.com/2010... )
- 'Mummi' Thorisson
Would certainly very much like to do that in the longer term. Keep it as a longer term target that any thing that gets built needs to be open to?
- Cameron Neylon
Agreed. Try and keep the concept of 'contribution' that one gets credit for (and can ideally be formally cited) quite open-ended.
- 'Mummi' Thorisson
First GDoc version: https://docs.google.com/documen... It's light on actual substance at the moment but that is kind of deliberate.
- Cameron Neylon
Can't edit. :)
- Andrew Lang
Ok, fixed the editing problem I think. I ticked the wrong box! :-/
- Cameron Neylon
What D said; I can't think of any substantive changes to suggest. Success will, of course, depend on picking the right people to be at the initial workshop...
- Bill Hooker
Absolutely - so phase II - who wants to be involved, who should be involved? Already some good suggestions above. This will necessarily be UK centric but hopefully not entirely so.
- Cameron Neylon
Just made some edits. I find two points missing: (1) If funders were to allow for public peer review of (at least some) proposals, the re-use of such proposals and reviews would provide a good use case, and probably be of interest to the funders. (2) As already mentioned above, wikis have solved much of the problem already (...
more...
- Daniel Mietchen
Agreed. But I would say this proposal is positioned to ask funders what they want first. Not for us to tell them what they should do. On the timeframe of this grant persuading any UK funder to do public peer review is a non-starter. In terms of Wikis, yes as a framework there is a lot there, but as you say the problem is you'd have to bring that content in because getting people to use them is difficult. But as you say the COASPED project would be a good example of the kind of thing that could be done.
- Cameron Neylon
So tldr: yes they're good points but they're kind of deliberately not the focus because the intention is to be open ended as to what will come out the other end.
- Cameron Neylon
Fair enough. I would like to be involved, but if it has to be via the UK, perhaps @science3point0 would be the best way to do that - would this counts as a "Infrastructure Organization"?
- Daniel Mietchen
Sorry for arriving late to the conversation, I've been dipping out, but I'd like to reaffirm Daniels sentiments and would happily offer S3.0 be It as a hosting service to the proof of principle or to act as an infrastructure organisation, either in it's current format or in a tailor made format for those involved, closed group etc
- science3point0
from iPhone
RT @neilhedley: Noticing a cool difference between FB and Twitter. FB is mostly friends I rarely get to see. Twitter is strangers I can't wait to meet.
Jan Aerts and Pierre Lindenbaum liked this
What has the public ever done for us...? - http://www.slideshare.net/Cameron...
 |
Pawel Szczesny, Bill Anderson, Khader Shameer and 17 other people liked this
Nice! 112 slides = 5 min talk?
- Andrew Lang
It was about twenty minutes. Trying to work out whether I can bothered cutting together a video and a screencast. Got about five minutes in but its really hard work...
- Cameron Neylon
You might consider doing a 'slidecast' instead: http://www.slideshare.net/faqs...
- Michael R. Bernstein
I kind of already have that in the screencast. The problem is that I tend to add a lot of material depending on audience response when giving these talks and it doesn't always make much sense out of context. And I find it difficult to find the time to do a fresh audio recording. I should probably just do that tho. Shortened version of the chat with the same set of slides.
- Cameron Neylon
Very useful! I'm sure there will be an opportunity for me to reuse this presentation! :-)
- Björn Brembs
The version on slideshare is only the PDF incidentally. If you want Keynote/ppt then drop me a line. Too big for slideshare...
- Cameron Neylon
I have to admit I quite like that sequence myself...
- Cameron Neylon
That was freaking awesome! And just what I needed as I head upstairs to talk about new work I might be doing which is supposed to be hidden so we can maximize profit...and this is TEACHING for crying out loud!
- Mickey Schafer
Mickey, I'm still trying to read your comment and my brain is failing to process it at all... :-/
- Cameron Neylon
Just talking off the top of my head, Cameron! Feel free to ignore the comment! It comes from a debate I am having with my supervisor on the availability of my own work. The argument began last semester when I refused to move my classes into a CMS which places everything behind a password.
- Mickey Schafer
...yeh that's what I mean. Failing to compute. The reason for moving your classes behind a password is because it will enable the school to make more money...?
- Cameron Neylon
No, the university makes no $$ that I know of -- it does allow some kind of data collection, I suppose, but mostly I think it's a means of creating scarcity where none need exist. The whole "scarcity increases value" concept. Fortunately, my supervisor does not fire me for disagreeing with him. Also, I've found lots of and lots of professors LIKE course management systems b/c it solves...
more...
- Mickey Schafer
Ok, the old chestnut. We spent money producing something, therefore it is valuable. Ergo we must lock it up so that no-one can ever use it...otherwise they might...you know...derive some value from it...
- Cameron Neylon
Prep. #10: the most boring control of them all - usually - http://bjoern.brembs.net/news...
Jan Aerts, Kubke, Sathishk and Daniel Mietchen liked this
Contains file with very raw data :-)
- Björn Brembs
Data tartare! The difference between the snapshot of actual raw data and the curated spreadsheet shows just how much work goes into making data interpretable and re-usable by anyone other than the original experimenter. I'm sure there will be an opportunity to use this example in a talk somewhere...
- Bill Hooker
rofl! :-) Seriously, though, I have most of the recorded traces (not all of them, though, unfortunately) and will make them available once I'm leaving here.
- Björn Brembs
If you add a few words of explanation to some of the traces, they might well serve educational purposes, e.g. on a wiki.
- Daniel Mietchen
Good suggestion! The traces are in the format of this nice little program, WinEDR. Will see what one can do with that...
- Björn Brembs
RT @Myrmecos: Count me as a big fan of the "graphical abstract", as per here http://www.sciencedirect.com/science...
Daniel Jurczak, Abhishek Tiwari, Jan Aerts and 2 other people liked this
Warning: Misusing the journal impact factor can damage your science! - http://cameronneylon.net/blog...
Mickey Kosloff, Heather, 'Mummi' Thorisson and 25 other people liked this
I had a bit of a rant at a Science Online London panel session on Saturday.As usual when discussing scientific publishing the dreaded issue of the Journal Impact Factor came up. While everyone complains about metrics I've found that people in general seem remarkably passive when it comes to challenging their use. Channeling Björn Brembs more than anything else I said something approximately like the following.
- Cameron Neylon
"...as professional measurers and analysts of the world we should be embarrassed to use JIFs to measure people and papers. It is quite simply bad science." Hear, hear!
- Bill Hooker
Totally OT - I would gladly pay to watch Aussies, Bill and Cameron play tennis (other games may be applicable) and all the proceeds be donated to a worthy cause. #anyonefortennis?
- Graham Steel
Ironically enough, just yesterday I filled in a form required for an application for a professorship, where they wanted to know how many papers I had in which IF journals. Should I be interviewed, this particular position would be so important for me, that for the first time ever, I would probably not say anything about this embarrassing use of the IF, which would normally disqualify them as employers immediately.
- Björn Brembs
Björn, was that Göteborg? I was trying to find the details, but they are apparently using that as part of the officiel job application process... did not find those details yet, though...
- Egon Willighagen
Now, and that makes it even more embarrassing, it was here in Berlin. When I interviewed in Uppsala I did not see any of this nonsense. If I get the professorship, you can be sure there'll be a lecture or two about IFs. And there will be figures with forms from certain universities...
- Björn Brembs
This whole sad discussion reminds me: why isn't there a tool available, that allows people to construct their own citation list??? I've been doing this by hand for years now: http://bjoern.brembs.net/citatio...
- Björn Brembs
Bjoern, have you tried PoP http://www.harzing.com/pop.htm?
- Bill Hooker
Yes, I use it, but it only pulls from GS which is neither as user friendly nor as 'accurate' as WoS/Scopus. So I use all three, de-duplicate by hand, copy and paste into an HTML editor and format (also by hand). I don't know of any other way to do this. To stay on topic: I think this is currently the best way to replace IF counts when evaluating people: use actual citations.
- Björn Brembs
In Poland JIF is used every single time scientists are evaluated (whether it's a grant or a new position). Also, quite often a lecturer on a science seminar is introduced with mentioning her/his "total IF points". We have also "ministry points" (from Ministry of Science). These are awarded in the same manner as IF - per publication (points are also awarded for writing a syllabus for a...
more...
- Pawel Szczesny
@Björn We're working on something with the intention of delivering this and PM-R has been arguing a lot recently for open citations and open metrics.
- Cameron Neylon
I would have thought that using JIF in a job application process would open an organization up to being sued...
- Cameron Neylon
@Pawel - that was kind of the basis of the prestige vs outcomes riff that I most recently wrote about in the interview with Michael. It's a perfectly reasonable decision for a country, particularly a small country to go for prestige as a way of making a mark. But they shouldn't expect that to lead to either a viable, stable, or particularly valuable research community. If you want those things then you need to optimise for them (which is harder to measure obviously, but most important things are)
- Cameron Neylon
I was thinking the other day about changing my cv and instead of listing 'my publications' start listing the papers that cite my papers (first order) and those that cite those first order papers (second order)) (or some quantification of that sort based on 'order'). A visualization of it could be fun to do too. Then I start wondering whether I should wait until I am out of my continuation period ....
- Kubke
@Kubke... agreed... if your research published in a low ranking journal but used significantly in Nature X publications, what JIF should you fairly take... should we perhaps make a black list of universities where JIFs are used? it seems that SHOUTING is the only way to get things changed these days... :(
- Egon Willighagen
@Egon :) I am on the advisory board for creative commons Aotearoa New Zealand, and one thing that came up is that 'opening up' requires a serious change in assessment policies. One example: Lets say someone gets 1000 citations on nature preceedings (not peer reviewed) shouldn't that count more than zero or 1 citation on a 'peer reviewed' nature? Should we move from 'peer reviewed' to 'peer accepted'?
- Kubke
Tres interesting, Kubke. >>> "Should we move from 'peer reviewed' to 'peer accepted'?"
- Graham Steel
@Graham or 'peer uptake'
- Kubke
And depending on who your peers are, we could have top peer, instead of top tier.
- Noel O'Boyle
What if the citing papers all cite the paper to dismiss it, or because it was shown to be fraudulent? You'd need either a citation typology or he possibility to retract papers from the record, the latter being difficult in non-peer-reviewed archives.
- Björn Brembs
@Cameron: Looking forward to that tool!
- Björn Brembs
@Björn It's not so much the tool. That's pretty trivial. It's getting hold of the data that is the problem.... but that's what the project is about.
- Cameron Neylon
Similar issue here as what Bjorn mentioned in the beginning: about to start a tenure-track, and one of the items on my checklist to be eligible for tenure in 5 yrs is "x papers/yr in a journal with IF >= y". Which obviously completely bypasses my open-source work... But at this point in my career there is nothing much that I can do.
- Jan Aerts
I think there are two things I would say to that. One is don't assume that tenure process in 5yrs will look like tenure today. Things are shifting, slowly admittedly, and perhaps too slowly but they are shifting. "Impact" and demonstrated income potential will be very important, both of which your prominence in the Open Source community will help with. Secondly, yes you need some good...
more...
- Cameron Neylon
@Björn wrt. citation typology: here's a recent paper on this very topic: Shotton. CiTO, the Citation Typing Ontology. Journal of Biomedical Semantics 2010, 1(Suppl 1):S6 http://dx.doi.org/10... "..ontology for describing the nature of reference citations in scientific research articles and other scholarly works, both to other such publications and also to Web information resources, and for publishing these descriptions on the Semantic Web. .."
- 'Mummi' Thorisson
@Mummi: nice! This sort of technology needs to be developed and incorporated in citation analyses are to progress.
- Björn Brembs
I favorited a YouTube video -- David Grady: The Conference Call http://www.youtube.com/watch...
  |
Jan Aerts and Rutger Vos liked this
Biosprint 2010 in Nairobi: http://rsg-ea-bio-sprint-2010....
Sonal Henson, Jan Aerts, Pierre Lindenbaum and Mike Chelen liked this
Nairobi... nice! BTW, where are all the scientists from Africa on the social web?
- Egon Willighagen
Seems not as many :( . Internet access and penetration has a role in this. Hopefully the situation will change.
- george
1st time out w/ my baby daughter only, she is six week old today, 1 step after generation Y
Yann Abraham, Khader Shameer, Graham Steel and 2 other people liked this
Consequences of gay marriage - http://graphjam.files.wordpress.com/2008...
 |
Mickey Kosloff, Björn Brembs, Chris Lasher and 7 other people liked this
I Hate Your Paper - The Scientist - Many say the peer review system is broken. Here’s how some journals are trying to fix it - http://www.the-scientist.com/2010...
Steve Koch, 'Mummi' Thorisson, Lit and 26 other people liked this
“When it comes to journals and publications, I’m highly skeptical that [the peer review] process adds much value at all,” adds Richard Smith, former editor of the British Medical Journal, who has written extensively about peer review. “In fact, it detracts value because it wastes a lot of time of a lot of people,” he says. “There’s lots of evidence of the downside of peer review, and very limited evidence of the upside.
- Alexey
from Bookmarklet
Wonder about studies that have measured the relative quality of various review processes?
- Mike Chelen
"The full podcast, originally created for the Journal of Participatory Medicine, is hosted here." http://www.patientpower.info/JoPM... (Podcast with transcripts) -- Peer Review and Reputation Systems: A Discussion -- 1. Defining the Problems and Issues with Peer Review Today -- 2. Light Versus Heavy Peer Review -- 3. Transparency in Peer Review -- 4. Wikipedia-Style Peer Review…and Rating/Reputation Systems -- 5. Crowdsourcing Research/Peer Review -- 6. Building a Community
- Claudia Koltzenburg
Hmm, I don't know if peer-review improves papers but it has so far often improved *my*papers...
- Björn Brembs
what's your take, Björn, does review quality depend on anonymity? should we say blind or non-blind should be up to the reviewer?
- Claudia Koltzenburg
btw, have not seen this idea voiced before ;-)
- Claudia Koltzenburg
I am with Bjoern here, I see improvement of papers on peer review (mine and others'). Both on substantial issues as much as smaller things. But i also don't think that peer review is a bulletproof system.
- Kubke
I think the point of peer review is not to improve papers but to prevent the publication of truly shitty ones. We really should judge the system on the true negatives. Kind of like democracy - democracy is great, not because it helps you choose great leaders, but because it helps you get rid of shitty ones.
- Bosco Ho
if publications were wiki documents then... +1
- Claudia Koltzenburg
Yes, like Kevin Kelly proposed (wikiscience): http://www.edge.org/3rd_cul... Additionally micro-contributions like comments and corrections should be measured and added to the overall reputation of a person.
- Konrad Förstner
I'm going to offer counter examples. I've had several papers that were damaged by peer review in my opinion and several more which were severely delayed to publication and not significantly improved by the process, leading to potential opportunity costs (one paper which took nearly two years to get published subsequently has got around 50-80 citations depending on what you count). but...
more...
- Cameron Neylon
Bosco, I disagree. Firstly the evidence suggests that most of the shitty papers get published anyway eventually and secondly, like I argued here (http://cameronneylon.net/blog...) actually the more rubbish is out there, the easier it will be to find what you want.
- Cameron Neylon
Bosco. While Cameron may be right on arguing against the papers part of your argument, I think the democracy part is a true gem......
- Nils Reinton
Yep Nils, I don't think search is going to do much to improve our parliamentarians unfortunately...
- Cameron Neylon
In both cases, though, bringing in transparency has lots of potential to improve the system. By the way, I listened to the podcast last night (not knowing about this thread then) and took my notes at http://ff.im/oVHcb .
- Daniel Mietchen
Not sure I've done this before, but -- I disagree with Cameron here. The assumption that more data = better search just doesn't convince me at all. I've seen a lot of papers that were basically trash, but that would appear as hits in any search that would also find good work on the same topic. I think the figure that gets bandied about is that only 30% of papers fail to find a home...
more...
- Bill Hooker
There's a step missing there. My argument was more that we need more data to build better search systems to enable better search. Putting out more stuff with today's Google won't help but it could help build tomorrows was the idea. In terms of the 70/30 figure the thing we don't have any real data on whether papers are improved, delayed, or made worse - most scientists say it improves...
more...
- Cameron Neylon
Well, we've had this part of the conversation too -- if reviewing were seen as part of a researcher's job and being good at it were properly rewarded, then studies like the BMJ one might not find such rampant slack-arsery. I guess part of this is failure of imagination compounded by lack of knowledge on my part -- I don't see how any search algorithm is going to be able to distinguish...
more...
- Bill Hooker
Imagine two papers, reporting the same experiments but in paper A the obvious control was omitted in each case, whereas paper B included proper controls. A and B may even draw the same conclusions! Both are going to be hits on any search I can imagine, and then you have to read 'em to know that A is rubbish. Scale that up, and you pretty quickly get beyond the point where you can read...
more...
- Bill Hooker
I begin to think that what we need most is more data...
- Bill Hooker
I don't really buy the opening crystallography example as an argument that peer review is broken. Okay, one reviewer suggested something unnecessary. There should have been more than one reviewer, and the editor (or his/her minions) should have reviewed the reviewer's comments, either omitting any blatantly ridiculous ones in correspondence back to the author or easily accepting the author's note that it did not need to be addressed.
- Rachel Walden
Bill, agree that more data is required to draw any sensible conclusions but in response to your point about paper A and B, as far as I'm aware we have absolutely no credible evidence that this is _not_ the case and some examples of dreadful peer review at the top of the pile. So should we be spending billions a year on something we have no evidence does any good?
- Cameron Neylon
In any case, my argument would be that unless you have both papers A and B then there is no way anyone can develop tools to distinguish between them, whether they be social or technical. Also surely the argument must be that both support the same conclusion and it is only by reading them together that you get the fullest possible picture...even without the control paper A strengthens paper B's case.
- Cameron Neylon
Bill: Why should the concept of search be limited to keywords? Consider Friendfeed for example, where results are highlighted by number of Likes. If anyone considers one paper better than the other, it would appear higher in the search results.
- Mike Chelen
Mike, may be a matter of semantics but when you get into "likes" etc, to me that's post-publication review -- in other words, a filter. I love the idea, but a glance at PLoS journals (and other experiments) will show that it hasn't taken off: people just don't interact with the research literature (yet?) in a way that makes social filtering effective.
- Bill Hooker
Cam, if A really is going to get published without the controls (70% of the time), then I'd have to conclude that peer review is a waste of time and money. I'm finding that hard to believe, but as you point out -- I have no evidence. I've made fun of other people for falling into the trap where "I can't imagine it" == "it's not possible", and now it seems I am hoist on my own petard......
more...
- Bill Hooker
A different way to consider peer review: it's as much a psychological barrier as a real one. So, consider paper A -- it may be shite but the authors thought it was good enough to pass peer review. If they didn't even have to consider that hurdle, what might they be pushing out? Is there a mountain of substandard work that would be dumped into the knowledge base but for fear of the Peer...
more...
- Bill Hooker
That argument I'd agree with more. One of the things I've not seen a discussion of is the extent to which the citation advantage in high IF journals is a result of author selection bias (i.e. authors are probably pretty well placed to say which of their papers will be most successful and send them to 'appropriate' journals on the basis of that). There's a flip side to this tho. What...
more...
- Cameron Neylon
from twhirl
see also http://journalology.blogspot.com/2010... (e.g., "CNS" disease explained) thx to http://ff.im/oZdka
- Claudia Koltzenburg
What is MapReduce good for? - http://petewarden.typepad.com/searchb...
Jan Aerts, Dan Hagon and Khader Shameer liked this
wish more research went into viz tools for collaboration & open source.. gource is pretty sweet: http://www.youtube.com/watch...
 |
Jan Aerts liked this
We are now open source! http://github.com/mhahnel...
Michael Nielsen, Sandeep Gautam, Daniel Mietchen and 6 other people liked this
Sweet: I'd go open source too, but I don't want you all to see how lazy I got while coding some modules (Very poor annotation in places). Congrats Mark!
- Brian Krueger - LabSpaces
I'm thinking of creating a development team. The 'board of advisors' you could say. Would anyone be interested?
Sandeep Gautam, Claudia Koltzenburg, Björn Brembs and 8 other people liked this
That would be a "Yes".
- Graham Steel
Sure, good thinking.
- joergkurtwegner
why not, but getting time commitments would be important.
- Sandeep Gautam
RT @Nick_Byrd: just discovered https://www.mendeley.com/. The weekend just got booked solid. #researchdatabase #nerdswillruletheworldsomeday
Jan Aerts liked this
Data science, roles, and barriers - http://mndoci.com/2010...
Mitsuteru N, Mr. Gunn and Jan Aerts liked this
PLoS Computational Biology: The Roots of Bioinformatics - http://www.ploscompbiol.org/article...
Mitsuteru N, Tom Walsh, Jan Aerts and Deepak Singh liked this
Other ways to read this feed:
