only 6700 updates? *o_O* - Yuvi

Yikes. Gnip to the rescue? - Alexandre Solleiro

that really means that xmpp/pubsub might be the future or at least some other model. - Rob Diana

How do you find these things out? - Brynn M Evans

6.700 updates/month or since Flick integration? - Benedikt Koehler

I think that's a misinterpretation. My understanding is that FF polled Flickr 2.7mil for 45k FF users, from which only 6k were actually online. I think there is a very big difference between the 2 messages. - Alex Popescu from twhirl

@Rob push models are not only the future, but the only real solution for such problems. - Alex Popescu from twhirl

@Alex you're more or less correct. However, those stats are from MONDAY -- a SINGLE day. The point is, FF made 2.7M requests to Flickr which returned less than 1% of actual updates. - Chris Messina

Andy's analysis is here http://waxy.org/2008... and the slide show is here http://www.slideshare.net/rabble... - Peter Dawson

Great presentation from Andy, excellent! - Jason Wehmhoener

that was Monday? wow that might not scale. - Rob Diana

@Chris I may be wrong, but it is a big difference between the 2 formulations. Anyways, the point is that poll model is not efficient and will start damaging the quality of some services. The only solution is moving towards push models (pubsub, xmpp), but this is quite a radical shift that some may not be prepared to accept. - Alex Popescu from twhirl

@Rob it will never scale. Just add a 0 to the number of users for which the query is made and the math leads us to 27mil requests/day from a single 3rd party. Another way: consider FF is gonna have to poll 10 similar services. It will have to deal with throwing out 27mil requests and process their responses which is equivalent with launching and processing 300+ responses/second (I've left aside the network latency and service response times). - Alex Popescu from twhirl

Alex, FF currentlly 42 service .and if hypothetically @ average of 27M per service, thats a lot of data to be pushed to a single consumption platform. Lets keep the user locked down to 45K. Any1 want to figure out capacity and bandwidth for such a scenario ? (Push !) - Peter Dawson

300+ responses/second is not really all that unusual in some situations (trading/betting comes to mind) - Jason Wehmhoener

@Alex, sorry, the sarcasm did not come through. I was trying to say that it will not scale long term, especially with how many users they currently have. Imagine the chaos if friendfeed went mainstream. - Rob Diana

@Rob Sorry, I haven't caught the sarcasm. But we do agree on the subject, which is always good. - Alex Popescu

@Peter I do think that we are saying the same thing :). Or is there any difference? Poll will never work at large scale. Full stop. - Alex Popescu

@alex, agreed !! - Peter Dawson

@Jason You're right. There are larger systems out there processing thousands of requests/s. But my point is that those numbers are only for gathering data. Then comes the data crunching + serving parts. Adding more machines/power may help but it is definitely not the solution. The only solution is pull, but for this companies would have to agree to consistently share the data in their systems. And that is a problem! - Alex Popescu

@Alex, "only solution is pull" - correct. Industrial strength applications work like that . By this, I mean Exchange data being transcation an fullfilled with certain set SLA'/ threshholds (OLTP spec's for Ecomm) and in near real time. However, both partners are charging their customers by transaction and there is a reveunue sharing on the model for both service providers. In the same analogy, Flickr , FF etc need to have a model to share data & revenue for it too become industrial strength. Why should Flickr share thier premium payers pics on FF for free ? or why should FF pull Flickr premimum payers pics ? IMHO, Justbecause the users wants it --does not make a good biz case ! - Peter Dawson

@Peter I'd say there is an important difference that should be underlined. In e-comm/financial services/etc they are handling private data, while Flickr/FF/YouTube represents "my" data. If I choose to put them online and pay for the service doesn't mean I am giving away my rights on that data. It should be "my" decision if I want it shared or not. Indeed, "you" (as in 'the service') can try to charge for this feature, but the 'data' is mine and I am the only one that should decide if it is sharable or not. - Alex Popescu

@Chris: Here is a link http://radar.oreilly.com/2008... that confirms my version of the initial message. - Alex Popescu