Tracy › Likes

Absolutely. I would also assume that Google is big enough that depts can easily keep secrets from one another. - Louis Gray

@Louis - big enough, perhaps. But easy? Not at all. Secret projects are rather antithetical to the dominant paradigm at Google. It's only happened on a few occasions that I can recall, and those were truly exceptional cases (that each had to be justified). Besides, the engineering culture favors a large shared codebase with a tremendous degree of code reuse and common tools -- more so than anywhere else I've worked -- it is hard to keep projects secret for long in such an environment. - DeWitt Clinton

Understood. I just don't always assume that the team working on GMail, for example, knows the Android product roadmap, for example. I know that certain more involved and visible folks, such as yourself or Mr. Cutts, may be more cross-functional, I would not ask Matt for updates on Blogger or Reader. - Louis Gray

Oh, certainly. And I wouldn't personally presume to speak for a team I wasn't working closely with. Though for most teams I bet all I would have to do is look at their team roadmap, which they'd likely share internally anyway. I think we've seen more value unlocked by having a culture of transparency and opportunistic collaboration (and to be honest, some drive-by critique) than we would by locking down too tightly. On the balance I know I much prefer it the way Google operates. - DeWitt Clinton

DeWitt's description is in line with my experience as well. With rare exceptions, if I don't know about a project it's because I just haven't heard about it, or haven't had time to look. It's a very, very open culture, and it's very rare to keep code hidden. - Joel Webber

This applies to experts in any field. - WorldofHiglet

It takes smart people to make complicated things simple. - imabonehead

Is it possible he's talking about Java programmers? - Gabe

i really liked this post (it resonated with me) until the end, at which point i felt alienated. - Neha Narula

What alienated you, Neha? To me, it seemed valid enough but a bit overwrought and trite. I know plenty of experienced, skilled working programmers who value just-get-it-done simplicity -- the "professional master" doesn't seem that elusive. - ⓞnor from Android

I'm a big fan of keeping it simple, but some problems do require a thorough approach. - Andrew C

"In contrast there are masters in the martial arts who learned their art as a means of survival and became masters in a realistic and hostile environment. We don't have anyone like this in the programming profession, " ... what about Carmack and Abrash & co? - Andrew C

BTW, I dunno if this is what put Neha off, but it almost sounds like Shaw wants to deny the reality of a nice O(n log n) solution beating out an O(n^2) solution (assuming small k, whatever) on a problem of decent size. - Andrew C

I mean, the stories of the martial arts masters may involve simple-looking moves, but they are also (in the stories) _perfectly_ executed, the product of careful observation of one's opponent and expert timing and precise angles. You might be able to pare down a simple linked list to the bare essentials, but I don't think it's quite analogous to not using a more complex structure _where appropriate_. - Andrew C

Nice... "The main thing I noticed about the experts I’ve encountered is they are into impressing you with their abilities. They are usually incredibly good, but their need for recognition gets in the way of mastery. Everything they do is an attempt to prove themselves and in order to do this they must perform like an actor on stage. There’s nothing wrong with this, and I don’t think the... more... - Ken Sheppardson

Andrew: Maybe the point was that an Expert would say "Aha! You need to keep these items in order, so a self-balancing tree is the perfect solution.", while a Master would say "Ah, but you never have more than 5 items, so a linked-list will always be faster!" - Gabe

this part, so much guy/son stuff! i dislike superfluous interfaces as much as anybody else: “There was this guy I worked with who once optimized a complicated red- black tree getting 300% performance boost. I was baffled and ask, 'How’d you do that? That’s impossible.’ To which he responded…” “'That’s my linked list my son.’” - Neha Narula

This is the kind of crap that gives java such a bad image. It used to be that people used it for what it was -- a simple OO language with garbage collection and a fast VM. Now you have architecture astronauts going off the deep end and making everyone assume the language has to be that way. I believe this disease stems from people who focus more on the process than on the product of their work. That's a recipe for disaster in my book. - Joel Webber from BuddyFeed

Neha: So lt's the fact that the language is male? - ⓞnor from Android

The impulse is good, but people have such different senses of what is simple, what has quality, what flows with the Tao. It's like beauty that way. What the story doesn't say is the 300% performance boost was on a limited test data set, in the real world it performed 3x worse and all the complexity had a reason that made sense once you "know." :-) - Todd Hoff

Complexity that's "there for a reason" is the worst kind. But who even talks about red-black trees vs linked lists? TreeMap vs LinkedList isn't the issue, interface swaddling and hyperfine dependency injection is the issue. Thing is, fights are decisively won, but code maintainability is much harder to measure, and even the importance of performance can be disputed. - ⓞnor from Android

I find it funny how the article, while praising simple approach, suffers from superfluity of language. - andrei_c

egnor: unnecessarily so. - Neha Narula

Neha, I thought the final "That's my linked list my son" was to make clear the parallel with the earlier quote "That was my foot my son" from Mestre Bimba. - Ruchira S. Datta

Todd: Imagine the situation where you are storing data for the US Census, and need to keep track of the people in a household by age. Since it's sorted and unbounded (there's no maximum number of children a family can have), you can easily think that a nice O(n lg n) algorithm that keeps a balanced binary tree is the right way to go. However, if you bother to look at the data, you'd see... more... - Gabe

I wish I could "Like" this article again :) - scott willeke

might have created a "MEGA-liked" button:) - alex melnikov

Love this article. - Mon Geslani

Search for Extra-Terrestrial Intelligence? - Gabe

Har har.... I assume there is ZERO chance of this ever happening. - Jay

@Jay - I suspect Paul wouldn't ask if he thought there was zero chance. Last time I remember Paul asking for something from Google to be open sourced it was their JS compiler. That took a while, but http://code.google.com/closure... - Nick Lothian

@Jay: Remember that Paul's referring to (relatively) generic infrastructure here, not search ranking code. But I think Daniel's right that it would be a *lot* of work, since most Google infrastructure is not "productized" and easy to wrap up in a bow for public release. Like any company with a lot of infrastructure, there are a lot of interdependencies that would be difficult to untangle. I think it would probably be better to simply publish papers on how it works, as with GFS, BigTable, etc. - Joel Webber

Boy, that would be a bold move Paul. Agreed that it would help out many though! - manielse (Mark Nielsen)

How about just dumping the source to the web without all the dependencies, even if it doesn't even compile? If it looks useful enough there's a good chance someone would adopt it. - Jim Norris

I suspect you're right about that, Jim. But Google would probably catch more crap about a "throwing it over the wall and letting it stagnate" open-sourcing than it's worth. But maybe I'm just down on it because Google catches crap no matter what these days... - Joel Webber

It's hard to open source distributed algorithms -- there's no obvious public standard to use, and the reasonable choices (TCP sockets? MPI?) are nothing like Google's internal infrastructure. I think a paper would be more useful than source code, the way MapReduce papers lead to Hadoop. Paul, have you looked at Vowpal Wabbit (http://hunch.net/~vw/)? It has experimental support for cluster parallelism, and I hear good things about it. - ⓞnor

Well, it doesn't have to be an either/or issue. - Jim Norris

If the code is too hard to separate from the infrastructure, then maybe a compute service like EC2 that provides an application interface specifically for solving problems with SETI could be good for both the world and good for the Google. - Bill Strathearn

@Bill: Now *that* sounds like a good idea to me, especially if accompanied by a paper describing the algorithms in use. - Joel Webber

Although technically really interesting and usefully, I do think that Google will not open source or even give inside information about such a key differentiating technology in the hands of their competitors. But I agree that it would be really great for the world. - yusuf arslan

I saw mention of JS open sourcing, did you folks see CoffeeScript yet? great stuff from Jashkenas, http://jashkenas.github.com/coffee-... - Mark Essel

The value of "differentiating technology" is not in novel algorithms, but in the thousands of places where implementations of these algorithms have been fixed and customized and tuned to solve the problem at hand -- which wouldn't have to be described in a whitepaper. - Tudor Bosman

@Tudor Bosman: I do not agree that the competitive advantage is the knowledge of fine tuning and implementing the algorithms. The concept/design of Google's machine learning infrastructure is very important. Don't get me wrong, I do think that Google SHOULD open source this. But I think they WILL not because of business considerations. - yusuf arslan

btw, ML+AI papers by Google - http://research.google.com/pubs... - http://research.google.com/pubs... - Amund Tveit

I think even just a paper would be very useful and fruitful. - Ruchira S. Datta

Ruchira, do you have particular biological datasets in mind? Are they too large even for a single machine version of Vowpal Wabbit to comfortably cope with (say, several tens or hundreds of billions of labeled instances)? - Simon

Simon, my comment was just general. Now that you mention it, I could use Vowpal Wabbit on some of my projects, which are not anywhere close to that big. Thanks for the tip! - Ruchira S. Datta

It's hard to get the taste of irony out of my mouth. - Kevin Fox

I really feel bad for these guys - Mehmet Aydin

iRan - Will Higgins™

I do to, and I'm sorry to even kid. I'm sure Twitter is being attacked constantly from various directions, from hackers to script kiddies to governments. I'm impressed they hold up as well as they do. - Kevin Fox

I think we all share Kevin's views. The timing of Ev's share was very unlucky. But nobody wants this kind of trouble. They have been stressed to the limit with growth, which in theory is a good problem to have, and they are maturing. Tonight was a rare exception to recent continued good news. - Louis Gray

Do the have a huge whale on top of their office? - Jemm

Hi Ruchira, as always your posts have been informative. Thanks - pradeep

Thanks pradeep. - Ruchira S. Datta

oh sweet http://www.tineye.com - too cool. Thanks! - See-ming Lee 李思明 SML

Useful app - Ozkan Altuner from iPhone

A movement sounds great. I support you! :) - Meryn Stol

As soon as Twitter opens up the firehose to everyone, someone can create a "mirror" of sorts that does all the right things PSHB-wise. - Eric Florenzano

Eric, I don't think they'll provide the firehose for free anytime soon. I think that right now it's an important source of revenue for them. - Meryn Stol

@Eric, are things like favorites in the firehose? What prompted this is that my favorites feed (http://twitter.com/favorit...) is still not updated on FriendFeed. All Twitter needs to do is stick a rel="hub" in there and run a hub (heck, they could use Superfeedr's or Google's) and sites like FriendFeed would get the fat pings instantly. A few hours of work and immediately the web would be better, faster, and more open. - DeWitt Clinton

I say - let's do it! - imabonehead

We can't... from the BirdDog content license - "5.ii.b - No Redistribution. Unless expressly authorized by Twitter, you may not distribute, sublicense, lease, rent or re-syndicate the Content or the Content Feed on a stand-alone basis, or display or perform the Content anywhere except on your Service." -- http://twitter.com/help... - Ken Sheppardson

This has already been discussed on the Twitter dev list (between myself and John K.) - short answer was no. Looking for that discussion... - Jesse Stay

Here it is - I was quite bummed by their response: http://groups.google.com/group... - Jesse Stay

Best solution right now is for us all to develop a standard that copies Twitter's, get that widely used and adopted with open source software that implements it, and then when Twitter is in the minority, tag on a real-time layer to it. - Jesse Stay

We (Superfeedr) are indeed working on that... it's not quite ready, but hopefully I'll have good news for early next week :) As a matter of facts, it works wuit well with user feeds already :) Search feeds are a little bit harder. - Julien

Julien, what are you guys doing about the terms of Service? I thought Twitter didn't allow that. - Jesse Stay

Per my link above, from John Kaluci: "Technically, someone could build a service to consume from the Streaming API and push into PubSubHubBub. This would be against the EULA though. " - Jesse Stay

"5.ii.b - No Redistribution. Unless expressly authorized by Twitter, you may not distribute, sublicense, lease, rent or re-syndicate the Content or the Content Feed on a stand-alone basis, or display or perform the Content anywhere except on your Service." -- http://twitter.com/help... - Ken Sheppardson

Here's my proposal - hoping you guys can join me: http://staynalive.com/article... - Jesse Stay

@Jesse -- good read. Just to frame this, I'll be honest, I'm not all that interested an "OpenTwitter." (Though I am all for a more open Twitter, and have nothing against people that want to clone the Twitter API.) I think what they've done is neat, but it is only a small part of what can be done once we make the web itself better at low-latency distribution of content + federation of identity and social graphs. - DeWitt Clinton

There's something much more powerful afoot than any single network or any single API. Think what could be enabled with Atom and RSS, PubSubHubbub (or rssCloud), Salmon, ActvityStreams, OAuth, OpenID, Webfinger, and Portable Contacts. That dwarfs any single thing we've seen thus far. In other words, don't think OpenTwitter; think bigger. - DeWitt Clinton

Seems to me the real attraction of Twitter's API is as a *publishing* protocol, not necessarily as a way to consume streams. For that we have the collection/stack DeWitt mentioned. But allowing someone to push out what they're doing form their iPhone, Air app, etc., sure... OpenTwitter's a good option. And that's how the platforms like Wordpress, Tumblr, etc are implementing it, right? - Ken Sheppardson

OpenTwitter could technically work on RSS. I'm thinking about a gateway of some sort that reads in RSS/RSSCloud/Atom/PSHB and publishes out in Twitter-compatible format so all the Twitter clients can understand it. I think it could actually work well with RSS, but for Twitter and others to adopt RSS, market forces are going to have to push them to do so. They're much more likely if we make it as easy as possible for the Twitter clients to do so. - Jesse Stay

* rssCloud insufficient as a base for mublogging federation. * we'll support it for real time updates (as much as we can). * if there's more functionality added to rssCloud / and/or other protocols that we can use, we'll support and adapt to new protocols as they evolve. Regarding: xmpp pubsub * it's heavy (need server, etc.) * we'd like to have it, maybe we should implement, but not a high priority right now. http://status.net/wiki... - A Mitchell

Thanks for the link to the OMB roadmap, A Mitchell. In general I like that direction -- identify and create the requisite underlying technologies, then build the special purpose (Twitter-like, microblogging, etc) services on top of them. - DeWitt Clinton

Also, I should add that I'm only suggesting rel="hub" and PSHB support for the per-user feeds. The firehose is a different story -- I'm not asking for that -- it scales differently and has direct implications for Twitter's business model. - DeWitt Clinton

There's some errors in there as well that I wanted to correct once I was at a desktop, like this glaring one: "Right now, GWT.runAsync does not seem to provide any hooks into the GWT environment to let the developer know, hey, the network is down." - Matt Mastracci

I thought your response was perfect. There are people who want more transparency and automation, ala Doloto/Volta, but I think absent super profiling compilers, expert systems, or machine learning, the developer is going to know best how to split up the application. - Ray Cromwell

"I notice that if you have the door to your office closed, you get more work done today and tomorrow, and you are more productive than most. But 10 years later somehow you don't quite know what problems are worth working on; all the hard work you do is sort of tangential in importance. He who works with the door open gets all kinds of interruptions, but he also occasionally gets clues... more... - Paul Buchheit

I've read this before and it has influenced me greatly. - Ruchira S. Datta

It has a lot of interesting insights. Long-term/short-term tradeoffs such as this door example are particularly interesting because people often get productivity advice that I suspect hurts them longer term. - Paul Buchheit

thank you, Paul, for sharing. I had a most inspirational bus-ride in snowy Estonia while reading this gem of a speech :) The avalanche of insights and concentration into 50 minutes is amazing and humbling. I'll definitely come back to Richard Hammings's speech to relive this perfect hour and concentrate on my dream of supporting creative process with software. - Baldur Kubo

" By realizing you have to use the system and studying how to get the system to do your work, you learn how to adapt the system to your desires. Or you can fight it steadily, as a small undeclared war, for the whole of your life." - Hayes Haugen