I, not a researcher, have to walk into a lab and make a stab at talking to the lab-folk about their data. If I am not to sound a priceless ass, I should know a little something-something about common practices and attitudes in that field before I make that walk. Whom am I to consult to learn that? If I ask the researchers, they'll think me a priceless ass and won't let me anywhere near... more... - D0r0th34

I am not sure that is possible. Heck even the lab people have trouble keeping up with the change. Unless we are talking about different types of data. Are you talking about data coming of instruments or the types of information that ends up at NCBI? - Deepak Singh

Either or both. You begin to see the difficulty. :) - D0r0th34

I'd argue instrument data does not have a role for librarians since you don't catalog that in the classic sense. The latter, I completely buy, and the question's a valid one. Also, I always wonder about the boundary between libraries and biocuration - Deepak Singh

Why don't we catalog it? :) And the boundary you mention is porous IMO, and I hope will become more so. I have a former library-school student embedded in this lab: http://www.loci.wisc.edu/ and she is doing fantastic work and they are very happy with her. - D0r0th34

1. the data changes every so often as instruments and techniques change. 2. Many times you are going to throw it away since you can regenerate it. 3. The data need to be reconstructed and it's the latter that are relevant. But I'll play devil's advocate. Most high scale data operations do not use librarians and are extremely effective with smart data management and good analytical tools - Deepak Singh

Sure thing. And they don't need the help. That still leaves a big bunch that do. - D0r0th34

You aren't convincing me at all :). I've worked in enough places where data classification has not been a problem or something that comes in the way. Now good data management on the other hand is another problem. Now if you're telling me that they are one and the same I'd like to understand how and why - Deepak Singh

It seems that in the end it should be the domain experts that finally deal with domain specific data - likely, utilizing and modifying generic tools/techniques. To what extent should a librarian become a domain expert? - Rajarshi Guha

Not necessarily so, Rajarshi. To what extent did geneticists become bioinformaticians? - Mr. Gunn

domain scientists, data scientists, computer scientists, and librarians < all living together happily... unfortunately, what we've got are domain scientists....... and librarians (of various flavors) who want to help - Christina Pikas

Christina, I still don't understand what you want to help with especially with "live data"? Sounds too much like a solution looking for a problem. Happy to be proven wrong. - Deepak Singh

Deepak, it sounds like you come from a very well-run lab environment. The stories I hear (some of which I have witnessed firsthand) are different: CD-ROMs stacked under people's desks, home basement servers, security-what-security, metadata-what-metadata, years of data left to transient grad students and postdocs to manage (documentation-what-documentation), zero sustainability plan past grant expiration or PI retirements... I could go on. This is an untidy state of affairs. Librarians abhor untidiness. :) - D0r0th34

I haven't been in academic labs in a long time :). I see two aspects. One is the archival and cataloging side, which is the library challnge, but even more what you are describing is a data management problem, i.e. a software challenge, not necessarily a library challenge with one constant, data types and data handling are going to change all the time. Of course, there's a third one ... just good practices. - Deepak Singh

All of the above, absolutely! The real challenge, though, is that decisions made early on in the process make a HUGE difference to the long-term preservability of the result. What falls out of that is that librarians can't sit passively by and wait for data to come to them; by that time, it's too late for a lot of it. So we're trying to figure out how to be useful earlier on. To do... more... - D0r0th34

Therein lies the problem. You can pretty much count on all your early decision to be wrong. Things are going to change and you can count on it. I completely get your point on the end state, but not on the data in process part. Libraries do not deal well with constant change and experimentation. - Deepak Singh

Libraries may not, Deepak, but wouldn't someone who knows a bit about organization systems have some useful input? I know as well as you how variable data types and formats are, but scientists aren't born knowing good practices to manage data in a changing environment. - Mr. Gunn

Perhaps. But I don't know any librarians who work in highly changing environments who are either, especially high throughput environments. We are not talking about repositories here. What you need are data scientists and engineers who are used to working in such environments. I am sure there is a lot of input librarians can play as data curators and custodians of the information and... more... - Deepak Singh

Agreed about primary analysis - Mr. Gunn

I don't think the fact that things are changing obviates a value for curation and good data management process. Software changes but the management procedures for best practice are built around that assumption, you still keep versioning your code though. The problem is most data production environments (on the small scale I work on) is that people never think about cataloguing, metadata... more... - Cameron Neylon

Deepak, I think you're missing a fundamental shift in where librarians contribute within the process. If you think of librarians only dealing with organizing, keeping, preserving *complete* work, then yeah, I see your points. However, in many places - including MPOW - librarians have been asked to contribute in finding information prior to data gathering and to help design and implement collaboration tools to be used during the data gathering and analysis process. (pt 1) - Christina Pikas