jeff hammerbacher › Comments

Wasn't the genome project kinda like this - open academic labs working on sequencing vs. Venter's private efforts? - Mr. Gunn

Not sure. The open project was still specifically funded, no? Was it open to all-comers? Wasn't there a technological restriction to entry, in that you had to have specific, new, equipment? A competition in an area that's more mainstream would be better. - Matthew Todd

I think about this stuff all the time; in fact, my last presentation (at WWW conference) references research from sociology of science. I am fascinated by some of the open science movements, but I think significant reform in the criteria used for professional advancement in academia will be required before we see a complete shift to openness. - jeff hammerbacher

Right, but no need to plea for change if we can demonstrate unequivocally that open research is faster/more efficient, and leads to what would normally be termed high impact publications. - Matthew Todd

Bring it on! - Chris Lasher

anyone have a petabyte-scale solution at their organization and care to comment on what vendor/technology they use? - Andrew Su

I guess the solution is in distributed databases... However for handling a large amount of data I've tried to use BerkeleyDB. It works fine but it tends to be slower for inserting data. Note: At #biohackathon2009, Jessica Severin introduced her technology named EdgeExpressDB (about to be published) - Pierre Lindenbaum

distributed database or filesystem, something like hadoop or tahoe should work, right? 6pb is "only" about $600,000 in drives at today's prices - Mike Chelen

i've seen several multi-petabyte data processing solutions built on hadoop: yahoo, facebook, quantcast are the public ones. - jeff hammerbacher

600,000 for the drives, what about the controllers and all the other infrastructure you might need? We were quoting 1 Pb for our storage and it was close to a 1,000,000. - Paulo Nuin

yes, surrounding infrastructure is also needed, they have 2 more years for prices to come down though right? facebook also uses a p2p distributed filesystem they developed called cassandra http://code.google.com/p... - Mike Chelen

Neil - This example came up (at a meetup in Waterloo, where both Ilya and I live) when Ilya pointed out an extremely nice feature this code has: to distribute the mapper function across the network, the code actually _sends the function_, as opposed to sending a text file containing the function. This is very nice because it allows you to use dynamically defined mappers. I'll use a similar idea in my distributed PageRank / MapReduce implementation. - Michael Nielsen

(Calling it MapReduce was certainly a bit... undescriptive, though, on the part of the post author.) - Michael Nielsen

Right - I did mean to look up those all important gems before commenting ;-) RingyDingy in particular: http://segment7.net/project.... Thanks for the info. - Neil Saunders

I spoke too soon. Looks like one can't pickle function objects in Python, which was how I intended to send the function over the network. Oh well. - Michael Nielsen

Michael, check out http://mail.python.org/piperma.... You can play tricks to pickle a function. - jeff hammerbacher