Great. Amusing that you have the Holy Bible as evidence in the toy chart. - Matthew Todd

Yes! it's an example from the 16th century, there weren't many refereed papers available :) http://www.evidencechart.com/charts... - Alex Holcombe

the most surprising thing I read about the heliocentric/geocentric debate is that in Galileo's time, supposedly noone brought up parsimony as a factor. So apparently people didn't consider all those epicycles of geocentrism to be a point against geocentrism. Even tho that's what I always assumed http://alexholcombe.wordpress.com/2008... - Alex Holcombe

Minor niggle - but the site gives the impression on the landing page that you need to login before you can see anything which is slightly offputting for someone just wanting to look around. Would be good to see the list of charts from the landing page so you can explore what's been done already. - Cameron Neylon

thanks Cameron. And unfortunately there's a lot more niggly user-interface imperfections beyond that one ;) we'll use your suggestion and others to improve it little by little - Alex Holcombe

Just thinking off the top of my head, for scientific issues, would it be interesting to ask the question, what papers are not included on this list? Again trying to tackle the selection bias you mention in the post... - Cameron Neylon

Since when do review papers tout theory? Maybe there is a class of them that do -- and perhaps this varies across scientific disciplines -- but the majority of reviews I read and allow students to use (and teach to write!) are NOT arguments supporting a theory -- they aren't even particularly biased. They do offer a personal perspective through selection of articles, organization of ideas, and naming of subsections -- and this certainly creates bias. But I've read very few (I read mostly in medically related areas) that are promoting a particular interpretation to the exclusion of others and still call themselves "reviews". - Mickey Schafer

"It allows one to create systematic reviews of a topic, without having to write many thousands of words, and without having to weave all the studies together with a narrative unified by a single theory." -- Alex, how have you created a systematic review if all you've done is categorized data? Or, simply gathered all the articles into one place? What's the basis for understanding one piece of research in terms of another if you do not have a particular framework within which to interpret? And what about the well-attested effect of writing on actual learning? I think an evidence-chart is a great idea -- like a concept map, such a device can help the individual keep track of what is going on. But a concept map succeeds b/c relationships are created between materials. The relationships do not have to be final, but they do need to be forged. The example provided looks solely at the single relationship of support, an interpretation itself that is well-attested to be influenced by individual preference despite evidence (take the bashers of acupuncture abounding on FF as an example -- despite evidence of measurable value as an adjunctive therapy and even the recent exposure of adenosine as the potential chemical underlying acupuncture's effect -- there are many still considering it part-voodoo. Evidence does not a believer make.) - Mickey Schafer

Alex, totally agree with you on the experience reading reviews as it relates to expertise. But I think the conclusion that reviews are thus biased is incorrect; expertise so changes perspective that relatively little one reads as an expert feels "unbiased" -- the science has to be phenomenally well done for us to be convinced. I read language stuff done by neuroscientists sometimes and find it frustratingly distant from what happens in fields of linguistics. But the bias in this case is mine. - Mickey Schafer

I absolutely agree that the requirement for the narrative format in review articles is as constraining as it is for communicating experimental results. Lets hope more people start experimenting with more modular formats such as this and share them on the open web. - Jean-Claude Bradley

From a discourse linguistic perspective, Jean-Claude, I find that statement really interesting. Maybe b/c I come from a humanities background way back when, I do not experience reviews as narratives, nor do I teach them that way. But I very much agree that the IMRD is narrative in feel b/c an idealized sequence is implied that may never have happened (and I tell this to students as well). Also, rather like the bible, the IMRD format was worked out consciously, decided upon by convention and decree. Still, there is incredible value for both science production and consumption for data to find its way into words, and to have the people who created the data tell what they think it means. I - Mickey Schafer

Mickey - I think that reviews are expected to be narratives in the sense that there is a "story" to be told. Reviewers will expect an historical perspective, key people involved, an exhaustive review of different aspects of the topic, etc. That is why it is annoying to read reviews in one's own field because we always find missing components. However just sharing information in a modular way as is shown here can be done because there are no such expectations. - Jean-Claude Bradley

Jean-Claude, it's the contention that one is "just sharing information in a modular way" that I'm disagreeing with -- not b/c it's a bad idea, but b/c it is not value-neutral. No chart is without a bias; it cannot exist without terms defining where entries go, so by definition, is not "merely" sharing. That would be the exchange of raw data; charts are a form of filtering. As for reviews, I read them somewhat frequently, but certainly outside my area of training. The particular species I teach, "state of the art" reviews, rarely contain long histories; are topic-driven, not people driven; and usually focus on a very specific area, which they do indeed go into exhaustively (ex: a paper I scanned last semester covering solely contemporary devices used to treat temporal mandibular teeth grinding -- perhaps something we could use right now;-)). - Mickey Schafer

this is an interesting organizational tool. I agree with Mickey - non-scientists may not find it useful and might event mock such an attempt to "pidgeonhole" ideas or see the format as a way to structure an argument to achieve a desired result. As a librarian with a lot of colleagues in the humanties and social sciences I am routinely surprised by how different I think about concepts or how I would make a case for a professional position. Invariably my colleagues get frustrated with me and tell me I'm not presenting all the nuances of a situation. - Elizabeth Brown

Mickey- I believe that any format will involve its own biases, so I didn't mean to imply that evidence charts are always better than reviews; just different in sometimes-useful ways. Any form of discourse has its own norms and perhaps even implicit assumptions. My own experience of writing articles is that to make it 'work' with the word limit provided and to get the 'story' to slot nicely into the reader's head, I am pushed to ignore certain difficulties and even to sweep things under the rug. - Alex Holcombe

Hi, Alex -- yep! I discuss these struggles with students all the time, and show them how it can change the interpretation from a reading point, too. The thesis students in particular struggle with this as PIs push them to condense 2 years of research into 10 pages or less. One student nearly cried 2 years ago when a single graph covering 1/2 page was the result of 2 years of life with 20 hrs a week spent in the lab. Even with a strong, positive result that had clinical application! Writing up science is often brutal work. - Mickey Schafer

Hi, Alex. I've just spent the afternoon reading several reviews written in various areas of biology. I think I must amend at least some of my previous comment: I read several papers that were clearly making arguments. This is quite different from what I usually read in medicine where a review is more, well, review-like. Could have something to do with the place I was reading (the "Trends" journals published by Cell Press, Elsevier). Overall, the writing was "bolder", more persuasive in tone, and interesting to read. But I can certainly see how some of the articles could be interpreted as way biased. - Mickey Schafer

Mickey yes, now that you mention it, it's also my impression that medical reviews tend to be more "just the facts, ma'am". I haven't been expert enough in a medical topic to judge whether they might somehow be just as biased, but being more subtle about it. - Alex Holcombe

Alex -- I am not qualified as a medical expert, but from a linguistic standpoint, I think the bias is there, but represented in the selection of materials rather than in how language is used. Frankly, I found much of the stuff I read today refreshing insofar as authors weren't hiding their agendas; I don't mind bias when the writer says "Look, I think this is what we should be doing and this is why". Also, I found many of these reviews included evidence charts of various kinds, much as you exemplified! But I don't have the background to know whether the categories charted were fair or not;-). - Mickey Schafer

Mickey- this is interesting; I have seen a lot of tables in papers, but rarely or never one with each column representing a theory and each row evidence. In a systematic review for example, each row might be a study and each column a methodological trait of the study, with other columns representing the results. If you could send me the references, I'm very interested to have a look at the ones you're referring to. - Alex Holcombe

I will need to go back to web site and see if I can locate a good example -- the "Trends" journals published by cellpress seem to require "boxes" of information -- usually 3-4 per 8 page pub. A few of those were charts. I'm not at a VPN enabled computer at the moment, so this may need to wait until Monday. - Mickey Schafer

This is awesome. I wish I knew it existed when I pushed my students to make this: http://pharmacomicrobiomics.com/papers... - Ramy Karam Aziz

Sure it is awesome, i like this idea, new vision and interesting! @ Dr. Ramy as u know when one do things from the ground up he learns more:), so it is still good experiment for me^_^ - len hat

Alex, here is a link to one paper that seemed to be using an evidence chart of some kind -- see Table 1, "Evolutionary hypotheses..." -- http://bit.ly/d15jVQ (sorry for the pdf -- wasn't sure if you had sub or not, and just seemed faster). - Mickey Schafer

@ Ramy, that is a nice education tool -- I have my undergrads do something similar, but use a Q/A format and have them answer in a bulleted list. Of course, my students then have to write those bulleted lists into prose, which is the point of the class for me -- it's also a great way to help students avoid unintentional plagiarism. Whether lists or charts, the information is distinct from the paper and students can write with clearer heads - Mickey Schafer