Hmmm... so what does this say about the "open science movement"? :-) - Bill Hooker

Sadly, Bill, it probably says that we're human ;-) - John Dupuis

lol :-) Good question. Probably one of the following: a) no rich corporation with political stakes in open science is funding us. b) there aren't enough scientific movements for one to be picked up by chance. c) open science lacks a certain something which would make it even remotely interesting for non-scientists. d) scientists are not yet numerous enough :-) - Björn Brembs from iPhone

Of course we're kooks in an echo chamber. It's just that we're right ;-) - Cameron Neylon

This is only dangerous if we continue to perpetuate the myth of "trusted sources". As long as we keep asking to see the raw data supporting claims (on the internet or otherwise) we should welcome a wide diversity of opinions. (And no - citing a peer-reviewed paper isn't enough - we have to insist on the actual raw data. Yes this is time-consuming but there aren't any shortcuts to truth. Without that debates will continue going in circles. When you chase down the actual data even wacky theories can get very interesting - a good example in chemistry would be hydrinos) - Jean-Claude Bradley

Point in case: http://friendfeed.com/neilfws... - Björn Brembs

@Jean-Claude: Agreed from one perspective. however, isn't it likely that especially these fringe groups will cherry-pick data which fit their views and everybody in this community will keep echoing that their views are supported by the data? - Björn Brembs

Bjorn - ALL scientific articles cherry-pick data. The question is can we explain the data using conventional theories. For example, hydrinos cannot exist according to current chemical theory. The lazy thing to do would be to dismiss all discussion. The responsible thing to do is to try to explain the results according to conventional theories. There are reports out there with enough experimental detail that they should be reproducible. http://tinyurl.com/ylhvxtm If you do repeat the experiments and get the same characterization data (e.g. NMR, IR) then that benefits science - and your theory of how to explain those results is also useful. There is nothing to gain in labeling people or theories as "fringe" - it is all just information and interesting whatever the truth may turn out to be. - Jean-Claude Bradley

@Jean-Claude - Some very nice points in there. Maybe we can discuss the point that ALL articles cherry-pick data at some later point, I'd claim that's somewhat of an exaggeration :-) 'Fringe', in my usage, mainly means 'very small fraction', correct me if this is wrong (but I am aware of the negative connotations). A crucial component of course has to be to recognize dangerous information, and this is where things can become very tricky. There is objectively false information which can threaten lives, just as speeding can kill people. As long as the veracity of information has not been sufficiently tested, I agree, it's all interesting. The problem is information that is definitely falsified, for instance 'MMR shots cause autism'. It's false information that kills people. Everyone knows a whole bunch of such pernicious memes. Fortunately, only few ever make it to the center stage. Should we do something about future dangerous information or is it just the sort of cost we will have to pay for the benefits of social media? - Björn Brembs

Speaking as a nonscientist, Bjorn: "do something about"? What? Make it illegal to publish falsified data? With who enforcing it? That's the price of freedom of speech, not social media. As for cherry-picking data: Is it possible to publish an article of less-than-absurd length that doesn't selectively report on the research, all its conditions and every speck of results, intended and otherwise? - Walt Crawford

@Walt: Those are exactly the questions I'm thinking about. Ideally, it would be a social system (since it's social media, duh! :) in which the proverbial 'silent' majority wouldn't be so silent any more. If such exceedingly small and objectively nuts but very active movements can garner traction, isn't it possible to think of a social STFU technology? In many countries in Europe, from analogous analog experience before WWII, if a political party can't get more than a certain percentage of votes, it won't enter parliament. In Germany, e.g. this value is 5%. However, political parties receive state-funds if they get more than 1% of votes. So there is a balance in that minority views are supported, but they can't wield real political power until they've reached some significant threshold, which usually takes quite some time and allows for proper vetting and testing. Is it completely inconceivable that an analogous technique can be found in the digital realm? I'm willing to live with downsides to technology, but I wouldn't want to suffer without having exhausted all technical possibilities of prevention - I do drive a car with airbags and seat-belts :-) - Björn Brembs

It's certainly conceivable that falsifiers could be shunned within given social subnetworks. Not sure that would have much effect in terms of gullible public acceptance, though--maybe even an unintended effect. ("See? Big Science is trying to Suppress The Truth!') Still, I guess, worth thinking about. - Walt Crawford

Bjorn - because of the requirements of communicating science via the article format you ALWAYS cherry-pick results. Take something as simple as the yield of a reaction. Lets say your postdoc does a reaction 3 times and gets 95%, 95% and 5% yield - what do you report as your actual yield? You probably cherry-pick the first two. Even though you can't figure out what went wrong in the 3rd run you ignore it because it is inconvenient and you don't have time to fully investigate. You don't mention the third run in the paper or supplementary materials. Then someone tries to repeat your experiment and gets 5% and calls you a liar. But if they had access to your lab notebook - with ALL experiments - they might be able to see something you didn't and report a caveat about the reaction. - Jean-Claude Bradley

@Jean Claude: I guess that depends really on the field and experiment, at least enough so to make the ALL and ALWAYS slightly overextended. In our field, we never have only 3 experiments of a kind, we have 15<n<40, none get left out, unless we can pinpoint the methodological error and then all experiments with that error get left out. If this is something non-trivial, this rule is in the methods section. This means that by now almost all experiments with positive results (and some of those with negative results) that I have ever made in my career are published in their entirety. Nothing has been cherry-picked. The remaining negative results could be published with no effect whatsoever on the results already published. This may be an exception, but one exception is sufficient to disprove ALL and ALWAYS :-) - Björn Brembs

Actually, not leaving out any data whatsoever is a mantra that is transmitted to all our students: If you have to select your data, do a different experiment. It's either all or none, no picking allowed. - Björn Brembs

So what would you do in the above case of 95%, 95% and 5% yield? - Jean-Claude Bradley

Depends on my experience with the experiment. a) Find the reason. b) Report a 65% yield. d) Repeat the experiment two (or more) more times and report the one 5% experiment as an outlier. e) If applicable, state that all yields below 6% were considered erroneous (if done correctly and among all experiments, this should not introduce a bias - if it does, it's not a viable option). - Björn Brembs

During my current experiments, it occurred to me that some details of the data have not always been visible for the readers in my last few publications. Due to the number of different experimental groups, we have been unable to show all the acquisition curves up until that final test that we do show. In one case I can think of, there may have been an interesting piece of information in there, that thus nobody but us here knows of. It doesn't change anything in the paper, it would just add more information. However, this information is a piece of something I want to have a separate publication on later, so not even this piece of data is cherry-picked - it's just not ready for publication, yet. I'm not saying this to sound holier than thou, just as one single example that the practices in different fields can vary quite dramatically. In our field, dropping the 5% result without a very good, solid, objective reason would count as unethical in our field. - Björn Brembs

Bjorn - there may be differences in fields. I don't know what the equivalent concept of the yield example is in your projects but when I did research in neuroscience as an undergrad I made the mistake of feeding the rats before a training session where food was used as a reward. I got yelled at - and I doubt those results were ever included in any paper. What if you were getting inconsistent results and found that one of the RAs was wearing perfume that interfered? Would you include those results in a paper? In the yield example most chemists would report 95% - as they should. When undergrads start in the lab they make mistakes you can't possibly track - and you will find lots of 5% yields for reactions that give 95% in the hands of a postdoc. I don't want to speak for your field but in chemistry every table is a lie - you don't ever really repeat reaction conditions exactly - it could be the room temp was 25C one day and 20C another, you're not using solvent from the same batch, the stirring wasn't exactly the same rate, the reaction was run for 100 min one day and 120 min the next, etc. If you had access to the notebooks (if they are well kept) you could see those small differences that don't make it in the paper - Jean-Claude Bradley

Jean-Claude, I wasn't arguing against opening the notebooks, of course (you know that)! :-) Feeding the animals (or incubating for too long or at the wrong temperature) is outside of the range specified in the methods, so not showing these data amounts to option e) above.There are whole papers on stuff like perfume interfering. I guess I'm trying to make the case that if there are non-trivial reasons for leaving some data out, either these reasons are in the methods or have their own publication(s), at least that's how it should be. Obviously, if the notebooks were open, everyone could also see which deviations from the methods did NOT cause the data to look any different! - Björn Brembs

Bjorn - coming back to the cherry picking here is a very concrete example. In this paper http://www.jove.com/index... we reported yields for 48 reactions run in triplicate and refer to EXP201, 202 and 203 as the source. However these experiments were run 10 times before that EXP189-200 http://usefulchem.wikispaces.com/All+Rea... We didn't refer to these previous runs because the results were not self-consistent and the control reaction was not coming out correctly. We had no idea why this was happening until the rep from Mettler-Toledo found a glitch in the code running the robot. Ever since I was a grad student I have referred to inconsistent results that cannot be explained as "the omega parameter". Sometimes experiments don't work for reasons that we don't have time to investigate and will never understand. These should not appear in traditional papers but I think should still be made available via ONS since someone else might see something that we missed. - Jean-Claude Bradley

Jean-Claude - great example and you're definitely right! Generalizing from this example, however, means that this is exactly not cherry-picking.What it means is: do the experiments in a different way and the results will be different. Of course, 'different' can mean anything and there is no way to cover an infinite parameter space. Some really smart people may actually benefit from such mishaps (and most will likely remain just that). What I think is a definite risk (which may be worth taking), is that in politically charged fields, you will have (for argument's sake) the anti-vaxxers latch onto that batch of experiments where your test-tube was contaminated, killing all your cells and tout that vaccines are deadly. The first time you hear of it, is when the main media are picking it up. By that time, there's no way for you to contain the information. Phil Jones received death threats and contemplated suicide in what I think is a very relevant, related case. I'm all for opening the books, as you know. I'm not entirely convinced, though, if opening them up to everybody is such a good idea (which doesn't mean I won't let myself be convinced it's worth the risk). - Björn Brembs

Bjorn I hear what you are saying - you don't want irrational people to take an isolated result and turn it into something you didn't say. But you can do that with any peer-reviewed paper you published. And certainly anyone can quote something you've said out of context from your blog or FF - but I don't get the impression that you are afraid to speak your mind :) The wonderful thing about this technology is that you can engage in the discussion if something irrational is derived from your work and it won't be hard to find your response with the way information is indexed these days. But censorship is not the answer. I think some forms of "trusted sources" are useful for filtering information on a subscription basis - as long as you can easily bypass them. I don't trust anyone to do my thinking for me although I appreciate different perspectives. - Jean-Claude Bradley

Jean-Claude - that's exactly what I mean. They're doing it already and opening it all up to everybody provides them with more fodder. I'm the first to admit that I don't know how to prevent that or at least slow it down. All I'm doing is trying to raise awareness and asking if something needs to be done. It worries me because I am convinced of the awesome potential of this technology - in both directions. Because it is so powerful and highly non-linear, the effects could be orders of magnitude worse than they are now or have ever been - precisely because they can also be orders of magnitude better. - Björn Brembs