publications and citations over time vs. lab size over time can give you a reasonable idea of where you're at - Michael Kuhn

This is a huge field of study in and of itself. Try poking around in the literature regarding the UK Research Assessment Exercise, for instance, or just google "bibliometrics". - Bill Hooker

Not "tenureology"? - Donnie Berkholz

I just realised that" tenureology" may not mean anything!! " Bibliometrics" may be a better measure - Aarthy

Thanks for the creative terms. I was wondering more in the direction of weekly lab output… rather than the "long term" view. - dK

Real-time microbibliometrics? - Mike Chelen

I think its a bad idea to think of traditional productivity metrics for most research professionals , i think the long term metrics serve their purpose and do some harm as it is - Hari

LOL. @Hari: this may be the case for academia, at least for those planning to stay there for some years. Industry? Different game altogether. - dK

I like the idea of meaningful actionable metrics, but is the number of papers a meaningful exercise? - Deepak Singh

How about papers/research dollar spent? - Björn Brembs

Including other project info like citations, word use, readership, file sizes or more can supply help to learn about and explore the research. Even if it is not a totally accurate measurement, the effort results in an improvement to the knowledge surrounding the lab or topic. - Mike Chelen

What I'd really like to see are some large-scale studies of research outcomes: how many drugs, crops, procedures, materials, etc etc do we have in 2008 that we didn't have in 1988 (or choose your timescale) as a result of research investment? How does that break down between public and private investment? - Bill Hooker

For the last decade, not sure we want to see the numbers (and they are available, at least for commercial investment $$$) - Deepak Singh

Bjorn is right, whatever metrics is used, it should be normalized over research budget. - Pawel Szczesny

Perhaps this is not what you are after, dekay, but I have been wondering for a while how to incorporate Ecological Footprint thinking in the assessment of research impact. Three papers relevant here are http://dx.doi.org/10... , http://dx.doi.org/10... and http://dx.doi.org/10... . A summary is at http://ways.org/en... . More in next comment. - Daniel Mietchen

Given that whole nations or corporations can have their ecological footprint estimated, I thought of extending this to research by looking at the resources used up for papers whose scientific impact is widely accepted to be high (one could take nobel-winning papers or highly-cited ones, for instance). I have had little encouraging feedback on this so far, and thus I have stopped working on it about two years ago but I would be glad to continue if any of you would like to join the endeavour. - Daniel Mietchen

I report the total number of experiments on our lab wiki on my annual report :) - Jean-Claude Bradley

You have to also remember to include your carbon footprint - as obviously - if you are a successful researcher you will be invited to lots of talks and keynotes. The larger the carbon footprint the more successful the researcher :) - Frank

@Daniel: what a wonderful idea. Have you checked out http://www.corporate.basf.com/en... -- at least for some industrial projects the footprint that is determined actually justifies research&development! - dK

@Jean-Claude: now, THIS is what I am talking about: what do you think, is the number of experiments performed a good indicator of research? How would you rate your experiments in terms of quality… etc? - dK

I suspect research efficiency is inversely proportional to the amount of effort invested into measuring research efficiency at any organization... - Eric Jain

Mathematician Stanislaw Ulam supposedly used to ask people who crowed too much about their achievements "What is this compared to E = mc^2?" When thinking about metrics, this always seems like a useful question to keep in mind - so little science is of lasting value. Not that intermediate stuff isn't valuable, too, but there are a lot of scientists out there with 300+ papers who history will rapidly forget. - Michael Nielsen

That's usually how it's measured, even the negative one's count, so even if you produce total garbage, which people will criticize inevitably: those citations still factor in. - sofarsoShawn

@dekay I put those experiments in my report and not all administrators weigh that contribution in the same way. I am lucky that some of my superiors do see value there. - Jean-Claude Bradley

@dekay quantifying progress is an elusive and, frankly, a dangerous thing to take too seriously. The intent of the researchers in a collaboration cannot really be measured easily but that is the strongest factor for the advancement of knowledge, in my opinion. We have used that criterion in our evaluation of the first Submeta Open Notebook Science Challenge Award winner today: http://usefulchem.blogspot.com/2008... - Jean-Claude Bradley

@dekay if you look at the criteria for evaluation it wasn't very quantitative but the judges reached a unanimous decision fairly quickly http://onschallenge.wikispaces.com/submeta... - Jean-Claude Bradley

Eventually, the sweet seduction of capitalism will bring the academics into the fold. Otherwise, who ever has the biggest library wins! - Jim Hardy

@J-C: Interesting. Nevertheless… we are on the journey to creating useful measures to improve the (meaningful results)/(€ spent)(time used). LOL. - dK

@dekay: Thanks for the pointer. I heard of that when I was visiting their plant two years ago but hadn't seen the link. - Daniel Mietchen

Some quote: "Hamilton [..] noticed that there were ‘people people’ and ‘things people’, classifying himself as the latter. Although these tendencies could be “disastrous socially”, he wrote, “I believe it is in essence an aberration of this kind that makes me a successful scientist”. " Source: http://www.nature.com/doifind... . - Daniel Mietchen

@Daniel: aargh. long live non-public-access sources. ;) - dK

Yes but the original source (quoted within the quote) was in a dead-tree book, so even less accessible. I paste it here for reference: Hamilton, W. D. in Narrow Roads of Gene Land Vol.3: Last Words. 206 (W. H. Freeman/Spektrum, 2005). - Daniel Mietchen