Starts with Albert Einstein: "not everything that can be counted counts, not everything that counts can be counted." - Shirley Wu

Who cares about impact? actually lots of people - public, funders, government, universities, etc - Shirley Wu

live video feed at http://www.mogulus.com/cameron... - Cameron Neylon

how is impact measured today?... with... Impact Factor! :-/ So what's the problem? over-interpretation. Shouldn't assign a paper the impact factor of the journal in which is is published. - Shirley Wu

other problems with IF: influences the publication process, citation behavior differs between fields, it's skewed(e.g. 89% of Nature's IF comes from 25% of its articles.), open to abuse and gaming, etc - Shirley Wu

Proposed alternative: article-level metrics. Ideal attributes: transparent methodology, repeatable / based on open data set, difficult to game, works across disciplines and publishers, be adopted by academia and reported by publishers, based on simple metrics - Shirley Wu

depressing anecdote: some departments require their faculty to publish in journals with IF of at least X. example of how IF is distorting peer review - Shirley Wu

So how can impact be measured? Potentially, where the work is published (traditional IF), citations (scholarly, hyperlinks, social bookmarks, etc), web usage, expert ratings (F1000, peer reviewers, ed boards), community ratings (Digging, commenting, rating), media/blog coverage (how to measure the authority?), policy development? (tangible changes), and more esoteric things like Who published it (institution, lab, previous work), who is talking about it + who is citing it (and what authority do they have?) - Shirley Wu

Web usage as an example. Some considerations: what to measure (if you download a PDF did you read it?)? who is the user? how to capture all usage data? how to compare? is it possible to game the system? - Shirley Wu

Adsense mention :). I am watching between calls - Deepak Singh

suggestion that the google adsense people have solved many of these problems - Cameron Neylon

but then good reasons why it isn't such a good model - Cameron Neylon

There are equivalents to PageRank in science, at least conceptually - Deepak Singh

audience also mentions similarity to Google's PageRank - Shirley Wu

EigenFactor is one. Eric Neumann also published a pagerank like system built on RDF - Deepak Singh

Has to be normalized to size of field as well. a 1000 views on a neuropsychology paper might be equivalent to 1000000 for a genetics of cancer paper - Deepak Singh

potential way to measure impact: "your article received x citations, viewed x times, received x comments, bookmarked x times, rated x by experts, discussed on x "respected" blog, appeared in x news media, etc etc (instead of single "your article was published in journal with IF of X" - Shirley Wu

Chris Patil: any new way of measuring impact is just a different way to substitute judgment? Is measuring impact even necessary/desirable? - Shirley Wu

authority 3.0. how to measure authority? laundry list of potential metrics. - Shirley Wu

Federated ID is essential, IMO and has to be built on top of an INTERNET standard, not some arbitrary scientific standard - Deepak Singh

Some audience discussion about researcher ID, social ID, etc - Shirley Wu

Some discussion about whether people (when searching for papers) look at impact factor before or after they find the papers, if at all, should impact factor come after making an effective search engine for papers? - Shirley Wu

Pedro shows plot of # of PDF downloads correlating with # of citations. Audience comments that # of citations does not indicate quality of paper necessarily. Maybe poor search engine led to crappy paper being read a lot, and you're likely to cite papers you've read even if they're not that good, because you didn't find anything better. - Shirley Wu

Which is why you need markup and have some form of discovery engine - Deepak Singh

Audience: why do we need IF??? some answers: hiring decisions, need a way to filter... etc - Shirley Wu

Goes back to the same point. What is relevance and relevance from what point of view - Deepak Singh

Kaitlin Thaney: impact/popularity != relevance to a researcher's problem - Shirley Wu

Bryan Bishop: we don't need a universal measure. Every search is biased towards the information you want to find. Subjective. Personal filters. - Shirley Wu

Johan Bollen is doing some research around this question: http://public.lanl.gov/jbollen...; http://public.lanl.gov/jbollen... - Hilary