Nor do I, but what he probably meant is to recognize that tenure committees are not willing to deal with much more than one number (let alone read articles or even check out wiki edits). - Daniel Mietchen

If you focus on that you will discourage people from repurposing their OA articles. - Jean-Claude Bradley

That's the core issue though ... measurement of scientific impact is more than just a single number. - Walter Jessen

Agree again, and think that education about metrics, as Heather pointed out, will be needed. - Daniel Mietchen

Researchers need to educate their committees in their applications - and include support from their colleagues - Jean-Claude Bradley

If you try to reduce to a single number how long before gaming the system with bots - Jean-Claude Bradley

not if it gets converted to a single number without people actually looking at the multiple values :) - Jean-Claude Bradley

Single numbers = onedimensional metrics. We could do this and call it the "number for the intellectually challenged": NIC ? :-) - Björn Brembs

In a scientific world where PCA is about the top of multivariate analysis... yes, NIC would do. - Egon Willighagen

Indeed. my point is that although a large basket of metrics is preferable, we live in the real world in which people want a single number, and without one they may never move away from the Impact Factor. Therefore, **perhaps** a single number (at the article level) would be a starting point in trying to change hearts and minds. I am certainly not saying that I think a single number is a good thing though, and rest assured we wont be creating one.It's up to the Academy what they do with the data... - Peter Binfield

Also note that in the real world tenure applicants need to digest and explain their accomplishments for the committee, which would certainly include any metrics, including IF. - Jean-Claude Bradley

@Peter... are there any plans to include citation numbers for the papers the paper for which the numbers are calculated? More valuable than blog counts, not? Details and context in my blog: http://chem-bla-ics.blogspot.com/2010... - Egon Willighagen

Not sure I understand the question - Peter Binfield

Say the citation pattern: C cites B, B cites A. Given metric for A... now #B papers are counted, but not #C papers.... if B uses functionality in A, and C cites B, then this is a partial cite of A... CDK use case in my blog: BRENDA enzyme database uses CDK functionality (fingerprint), BRENDA is cited 241 times, CDK itself a mere 115 times... Should the CDK papers not inherited a weighted bit of the BRENDA citations? Now, PLoS ONE does not weigh anyway, so why not report the number of papers that cite papers that cite the paper for which the metric is calculated? - Egon Willighagen

Ah, I see. We take all our citation data direct from 3rd parties (Scopus, CrossRef, PMC) so unless they have that data, and can provide it easily, then we couldnt have it either. Also, have you ever looked at the methodology behind Eigenfactor ? - Peter Binfield

When they just come into existence I did... but from my memory, it was mostly oriented at journal impact, not article impact... that changed? Need to check it out again then... - Egon Willighagen

Nope, I think it is still journal level, but they incorporate the similar concept of weighting based on where the citations are coming from. At the end of the day it is a massive network analysis problem which I assume would get exponentially harder when you move from the journal to the article level. - Peter Binfield

Not sure... it's just a set of linear equations... finding the solution is iterative, and they can stop at any convergence criterion they like... probably, very much like PageRank... - Egon Willighagen

We covered the Eigenfactor in my cheminfo class - see FAQ16 http://getcheminfo.wikispaces.com/FAQ - Jean-Claude Bradley

Egon, isn't what you are talking about here effectively page rank for papers? You estimate influence by traversing the network. Being cited in e.g. one BRENDA paper makes a small difference to overall rank but being cited by even a relatively small (~10) of highly cited papers has a much bigger effect than simply having lots of citations from uncited papers. - Cameron Neylon

Yes, quite so. I guess anyone can do this, but it depends on the availability of public citation networks... - Egon Willighagen

Yes, public data is a big problem. Which is lucky since I just happen to have a proposal to build federated repositories of citation data in my back pocket if anyone would like to fund it.... - Cameron Neylon

Laughing at Cameron's last comment, because I know it's true. Wish I had the money, Cameron. - Jason Hoyt

It's alright Jason, you guys are on my list of people to target to help us get it funded... - Cameron Neylon

The German Research Foundation (DFG) this week announced that starting in July they only want to see the 5 best papers in a CV for a grant application. Very big step forward. - Martin Fenner from iPhone

@mfenner wondering what their definition is of "best" - suelibrarian

@ suelibrarian, they phrase it as something like "those most pertinent to the research proposed", which reads between the lines as basically those containing the "preliminary data". Anyway, the policy may well have been in place last July already, and I followed it when submitting a proposal in November. - Daniel Mietchen

I think we need as many numbers as there are concepts being measured. "Impact" is definitely multi-dimensional, so we need multiple dimensions. That said, probably not 15 dimensions (but probably yes 15 (or more!) datapoints to inform the several dimensions). By understanding more what the different dimensions are and why they matter, we can help tenure committees etc understand which ones they should care about. - Heather Piwowar

Jean-Claude has a great point about openness early and often decreasing the traditional "impact" metrics, because the impact is spread across preprints and articles for different audiences. Ideally we come up with metrics that reflect this. somehow. :) - Heather Piwowar

when I served on our university personnel committee for faculty a few years ago, we NEVER discussed metrics while deliberating on renewal/tenure cases (impact factor, number of citations, etc.), nor was it a factor for decision-making. Has this suddenly changed in academia and now metrics are really important? - Elizabeth Brown

Thanks, Dorthea, I do remember that discussion now. I see more interest/awareness of metrics in the sciences here, not necessarily as a tenure tool but more as a personal tool to gauge impact of your work. Some faculty count citations, but I don't see much interest in impact factors or alternative metrics. I haven't seen lots of evidence that US funding agencies are using IF or quantitative metrics for decision-making. The one NSF grant application I was involved with recently didn't ask for any metrics. - Elizabeth Brown

During my last participation in a tenure committee we did not explicitly discuss IF, although the candidate did use the IF as an argument to show the quality of his publication record. - Jean-Claude Bradley

The VIB institute in Belgium has periodical goals for their PIs where they have to meet a certain number of publications in specific IFs .. The sort of X papers above 5 IF , Y papers above 10IF. In my home country (Portugal) it used to be the total number of papers which lead people to collect stamps but is is shifting now to citations and IFs. One good example in europe appears to be the ERC. So far I have heard good things about the evaluation process although there have been way too many applicants for the available money. - Pedro Beltrao