Cameron Neylon
A collaborative proposal on research metrics -
When we talk about open research practice, more efficient research communication, wider diversity of publication we always come up against the same problem. What's in it for the jobbing scientist? This is so prevalent that it has been reformulated at "Singh's Law" (by analogy with Godwin's law) that any discussion of research practice will inevitably end when someone brings up career advancement or tenure. The question is what do we actually do about this? n opportunity has arisen for some funding to support a project here. My proposal is to bring a relevant group of stakeholders together; funders, technologists, scientists, adminstrators, media, publishers, and aggregators, to identify needs and then to actually build some things. - Cameron Neylon
Wonder whether Andrew Treolar might have insight to contribute - Kubke
AT certainly has experience of the problem and also seems to be collecting a lot of data... - Cameron Neylon
He was strong about making data sets a primary citable object at the data matters meeting where I met him, to make the 'use' of the data as trackable as any publication - Kubke
Yep, I think that's a key theme. Need to make sure have Dryad and Datacite involvement with this. - Cameron Neylon
He suggested giving the 'data' a DOI and tap onto the search/link/tracking that already exists for paper publications as something which could provide an initial viable solution - Kubke
Essentially this is what the DataCite project are doing. I think Dryad is connected with them as well. I worry about using dois for this because I would rather URLs [ducks in case geoff bilder arrives to pummel me (-; ] but the reality is that the scientific mindset seems to be doi = real so I think it's going to be the way we move forward in practice. - Cameron Neylon
The redirection mainly. Essentially it breaks a whole set of fun things that you can do with URLs. None of which to be fair are very commonly done. Also in principle they could be enabled through the doi redirect. Essentially its a minor quibble that it doesn't add anything technically and breaks some stuff. But the reasons for dois are fundamentally social not technical. - Cameron Neylon from twhirl
I'm a big DOI fan. Just yesterday I ran into a few duplicate papers in the Mendeley library. One reason is that there are several URLs associated with a paper: PubMed, Journal HTML page, Journal PDF, institutional repository URL,... Not to mention that many URLs break over time. - Martin Fenner
And DOIs would really help to automatically find science blog posts associated with a paper. The Journal of Neuroscience editorial about supplementary information in August has no DOI. Almost impossible to find all blog posts (there are many) that talk about this editorial. - Martin Fenner
All true but I guess my problem is that fundamentally dois are a special system so that generic web tools (and yes particularly semantic web tools) will break over them and the info won't get incorporated into the wider web. e.g. Martin's problem could be solved by an owl:sameAs which would mean that it would be transparent to other web systems. But with dois someone has to build something special to link that in. I think my other objection is that it perpetuates the idea that nothing is real unless its a journal article. And real = journal article = doi - Cameron Neylon
None of which changes the reality. DOIs are dominant and are here to stay as far as I can tell so I just need to get over it and move on... - Cameron Neylon
Just added a comment and we need to do more than offering incentives for "re-use". - joergkurtwegner
While I am a big fan of finer-grained measures of contribution (e.g. peer review), I am not convinced that the current proposal should be stretched to cover such measures. It seems to me that what is wanted here is a hard-nosed, outcome-focused project designed to answer hard questions about ROI. For that purpose, a close focus on re-use is probably optimal. - Bill Hooker
Nice proposal. It currently focuses strongly on funder needs. Useful, perhaps necessary, but it means delayed and indirect rewards to the jobbing scientist. Would it make sense to work in a more direct focus on "jobbing scientist needs" too, with possible hacks of automated reuse-metric CVs and reuse-metricful tenure packages? - Heather Piwowar
Joerg - I commented on your comment back at the proposal. Two things really. My view is that the "measuring re-use" idea covers things like review. A good review needs to be "cited" in some way and that re-use measured and rewarded. I'm not quite sure where you're going with the micropayments thing. A kind of micropayments has been tried by EPSRC in the UK where they pay £50 or something into departmental accounts for each on time grant review. At best this is an irrelevance as the amounts are too small to be useful and at worst an insult. My time costs out at around £200 an hour. Doing a proper review is somewhere betwen £500 and £1500. The problem to me is that micropayments just doesn't cut it, these are macropayments. Now on the other hand if we got a small amount each time the review was used then I'd be intersted, but then we're back to measuring re-use again aren't we? Am I missing something here? - Cameron Neylon
Heather, good point. Although the response I've got suggests that people are interested enough in the possibility of getting credit for a more diverse range of things that thats enough at the moment. But yes, the CV is a very good place to realise some benefits. Need to talk to someone inside VIVO I think to see if there is some low hanging fruit there. - Cameron Neylon
@Bill , @Cameron - I understand your views and the ROI question is important. As long as we agree that there is some measure, recognition, and reward scheme I agree. So, a re-use needs to be tracked otherwise, we need to prevent ab-use of re-use. - joergkurtwegner
So if I'm reading you right your core point is that we shouldn't assume that simply by measuring something that the rewards will flow from that. That it is important to consider explicit reward schemes, which might include payment or aggregated micropayment? - Cameron Neylon
Actually it strikes me that both Joerg and Heather are pointing in the same direction here. Keeping up the issue of direct benefits. - Cameron Neylon
Cameron, what is your timeline for the proposal and what sort of additional help would be useful at this point? The current draft doesn't have references: do you need any? Other help? - Heather Piwowar
References (both to literature and to other projects) would be a big help. Contacts with funders would be useful if anyone has them. And proof reading still valuable. I'm currently checking with the funder whether this is heading in the right direction and then I'll start pulling in contributors more. Timeline is really about a week though the quicker the better I suspect. - Cameron Neylon
Also anyone got a good contact on the inside of VIVO? I know that Mackenzie Smith is on their advisory board but don't have any good contacts on the inside. It would be an obvious source of data. - Cameron Neylon
Sounds good. I've added (REF) indications where it seems like references might be appropriate, to facilitate crowdsourcing. Cameron, please edit as needed based on the intended document length and number of refs expected by the funding body? - Heather Piwowar
Yep, still working on those last two questions... ;-) - Cameron Neylon
what came of this? - Claudia Koltzenburg
Got the money, running the workshop: - Cameron Neylon