  |
May 30 at 5:07 am
- Link
Lucretia Pruitt, Evan Sims, Joshua Smith and 28 other people liked this
I twittered about twitter lame excuse ~5 hr ago...they can't be serious. - Tal Keinan
The comments are really pissing me off. FriendFeed is 1000 times more reliable. Twitter was going down before I even got popular on the service. Their architecture has always sucked and everyone knows it. They've never been able to get a handle on the quality of their service and now it looks like they are blaming their top users. Wonderful. I'll just go to the service that CAN keep up with my load. So far FriendFeed has been. - Robert Scoble
*face palm* - Andrew Dobrow
Have I missed something? Have Twitter actually blamed you (and other "super-users" for their problems? (UPDATE - Ahh, scratch that, just found the post). - Ian Betteridge
Ian, yes, they did in their latest technical post. Not using my name, but they said that noisy users with lots of followers are part of what's causing lots of load on their systems. - Robert Scoble
VentureBeat post in question http://venturebeat.com/2008/05... - Andrew Dobrow
Twitters excuses are pathetic. If one power user could take down Twitter, they should stop their services. Okay, Robert Scoble is a real power user. But if a service can't handle one of those guys, it shouldn't be on the web. - Ryo
Time for the "noisy users" to take their conversations elsewhere? I wonder where they could go that would support them and be able to handle the traffic they bring with them....hmmm. - cmiper
Ridiculous. It is NEVER the community's fault. And it's NEVER ok to blame the community. I missed this excerpt on my initial reading of the post on dev.twitter. Even if the technical basis is because of some sort of data pileup, that's because of their shitty scaling and their lack of planning from public launch. They should be able to handle a rapid succession of actions if they had their head screwed on straight. - Andrew Dobrow
OK, so this is what Payne actually said: "The events that hit our system the hardest are generally when “popular” users - that is, users with large numbers of followers and people they’re following - perform a number of actions in rapid succession. This usually results in a number of big queries that pile up in our database(s). Not running scripts to follow thousands of users at a time would be a help, but that’s behavior we have to limit on our side." - Ian Betteridge
Agreed - It's all BS to blame you for their technical issues. Maybe we revisit the pay service debate? It's only going to grow. Twitter has to have the ability to keep the fire going. - Lisa Thompson
And even if this IS the case. The fact that Twitter didn't plan for this by having some sort of friend limit is THEIR administrative absent-mindedness. - Andrew Dobrow
My question is, isn't FriendFeed sustaining the same impact when we all get on here and 'chat'? I've never had a problem with FriendFeed, and most especially never got the annoying pic of the birds carrying a whale..... - Paula Hawk
Robert, Twitter has recently been pretty open about what its architectural problems are lately. The way I read what Payne is saying isn't that he's blaming "popular" users: he's explaining, openly, where the system is not coping as well as it should. As for how FriendFeed does it... think of the network effect. Think of how the number of interconnections times the number of users affects the number of total database queries. You can work it out :) - Ian Betteridge
To be fair a) Friendfeed could hit the same problems if it becomes as popular and b) what they are saying may actually be true. This may be the cause of the database jams. This is reasonably unchartered territory and technical hitches like this will be discovered. It's like heating chemicals to extreme temperatures - sometimes strange shit happens. - Martin Weller
Possibly the lamest excuse for chronic system unworthiness in the history of social media. - Chris Baskind
Paula: I believe the traffic levels here are far more than anything Twitter saw in its early days and it's WAY faster and WAY more reliable than Twitter was at its same stage in life. - Robert Scoble
I don't think it was intended to get that across as an excuse... perhaps he was only trying to say what causes a problem. (UPDATE - although, it does seem like a serious design issue if they didn't consider such loads upfront.) - Parth Awasthi
Gotta say, if they're blaming the people with large follower/following numbers, then they just admitted to have a serious problem with their architecture. If you can't handle people USING your service, well then you shouldn't be in the service providing business. - Michael Koby
@Parth But if it does cause a problem it is not anyones fault but their own. If they weren't prepared for this type of growth and this level of scaling, there should have been a friend limit, much like Facebook has. - Andrew Dobrow
Robert - would it be churlish to suggest a week of you, Leo, Kevin R etc dumping as many tweets as you can into twitter? ;-) - Mat
This is the problem with being transparent about issues like this: someone picks up a quote, headlines it with "Twitter says it's the user's fault" and everyone froths about it. - Ian Betteridge
Martin: I don't think FriendFeed will hit the same troubles. First of all, FriendFeed is growing MUCH faster than Twitter ever did. It took me more than a year to get more than 10,000 followers on Twitter, while I did that here in only a couple of months. Second of all, FriendFeed's team understands how to build scalable systems a lot more than anyone on Twitter's team does. You HAVE to look at who is on the team. That's why I chose this over SocialThing or Profilactic. - Robert Scoble
Darn noisy tweeters ruining it for everyone. j/k :P Pretty lame-o excuse for not have the architecture to scale properly. - Mathew A. Koeneker
Michael: Twitter already talked about their architectural problems: http://dev.twitter.com/2008/05... - Ian Betteridge
Robert, I'm favoring FF more and more as you do, and Twitter is only helping FF right now. - Brian Carter
Ian: no, the answer is to be MORE transparent with issues like this and they should be participating in the conversation, which they aren't and rarely have on Twitter, even. That's a MAJOR part of the problem over there. No one on Twitter is really a serious Twitter user. I look over here at the FriendFeed team and I see tons of serious FriendFeed use. - Robert Scoble
I bet you Scoble is blogging atm. Or at least planning a post soon :-D - Andrew Dobrow
this should be interesting to watch! - ben rogers
via twhirl
The Google guys are way more prepared than the Twitter guys. See my comments at http://friendfeed.com/e/03473e... I used to make $billion apps in the 1980s, and it was laughable to see the early Internet crashes on the big Amazon & eBay systems. Even Meg Whitman at one point admitted that she had to rethink her tech strategy for eBay - that better reliability required a whole different approach. Welcome to the world of BIG stuff! - Mitchell Tsai
I think a lot of you are taking Alex's comments wrong. He's not necessarily saying "power users are bringing down Twitter". What he's *actually* saying (at least how it looks to me) is that Twitter's architecture was built wrong, and the way it was built makes it easy for power users to put a huge strain on the system. While it probably sounds like they're placing the blame on power users, they're actually placing the blame squarely on the poorly-constructed Twitter architecture. - Nathaniel Payne
I would not even care about Twitter if you hadn't been talking about it. They need some perspective. - Josh Mings
have to agree with Robert on this one - they should be active on twitter - watching the has tags and commenting where necessary like brightkite and zappos and seesmic etc etc - ben rogers
via twhirl
Yep FriendFeed is working so great Robert's video doesn't even work and nobody that knows the nature of social networks is really blaming Twitter for going through what is typical for any social network and that includes FriendFeed. To date Twitter has been the most reliable for the number of users and the allowed tweets. Now Twitter spammers and the many numbers of folks who join your feeds and then find they have the time to read their post need to get off but super-users like Scoble, Laporte and Rose Don't need to leave. Obama, McCain and Clinton if she's on do...nobody is reporting anything on Twitter the cannot find on the news an hour ago. - James Bess
My site went down earlier for about a half hour and it's Scoble's fault. - Paul Short
Robert, nothing in the original post says "Blame Scoble", or even "Blame the users" - and yet that's what you're claiming they're saying. They explain what the issue is clearly and concisely, and yet now you're leading the crowd with the pitchforks and posting what's clearly an angry video clip. Calm down, mate. At the end of the day, if you don't believe their service can cope with what you want to do with it, change service. - Ian Betteridge
Whether or not Twitter's problems are due to "noisy" users with big followings is immaterial. The fact is they are backed into a corner and are turning to "transparency" as a last resort to try and buy some time. Unfortunately for them their 15 minutes of fame has clearly passed. If this is really the core problem with the recent outages it is a problem that could or should have been anticipated and dealt with a long time ago. That it hasn't is testimony to a lack of attention at Twitter by the management/founders. I suspect though that the problems of architecture, technology and personnel run much deeper. The people whose $15 million was recently invested I think can kiss their money goodbye. - Brian Sullivan
twas a bizarre call for them to use that excuse. - Darren Rowse
James: what are you talking about. The video is working great and so is FriendFeed. Twitter has NEVER been reliable. - Robert Scoble
Ian: read the headline. When you allow journalists to use headlines like that, and you don't get involved in the comments, then you are condoning them. Also, the engineer clearly blamed "noisy" users like me for causing stress on their systems. Well, I'm 100x more noisy over here on FriendFeed and I have about half the number of followers here already. Why isn't FriendFeed seeing ANY slowdowns? - Robert Scoble
Ian: The article states "Most users of Twitter will single out one person who this points to: blogger and Fast Company employee Robert Scoble." That's pretty blatant in the conventional-business world of "play pretty" & "politically safe". I think Robert has a good right to be upset! - Mitchell Tsai
Sometimes the worst thing that can happen to someone/something is that it gets wildly popular. Back in the day podcasters would get a budget hosting service, get popular by some freak of nature, and crash servers and eat bandwidth. Didn't Twitter get some funding? Was that to cover pork rinds and beer? HIRE SOMEONE and get a better backend. - Lynette Young
Oh come on Robert. "Allow journalists to use headlines like that"? Thankfully, we don't live in countries where companies can determine the headlines about them. And no, Payne was not *blaming* noisy users. He was making a clear, matter-of-fact statement about the issues the system has with noisy users. As for why FF works well... well how many users total does it have? How many does it have that follow/are followed by over 2,000 others? Network effects count here - the more large users, the bigger strain. - Ian Betteridge
It doesn't even matter if they aren't blaming Scoble or if they're blaming Calacanis, or if even they're blaming Henry Rollins for all I care. You don't ever blame your own community for causing the problem. Ever. Even indirectly. - Andrew Dobrow
I lost a scoop of ice cream off my cone -- it's Scoble's fault cuz he Twitters too much :\ - Shey
Andrew: That statement does not blame anyone. It just says what the issue is. You want them to lie, and so "oh yeah, our system is totally happy with people doing this, it needs no more work than a few hundred more servers". - Ian Betteridge
From a technical perspective, most people don't design "scalable stuff" because they've never seen the issues (either from academic research, or practical industry). I don't know the inside issues, but from Scoble's description of the performance of Twitter and FriendFeed in the their early days, I'd guess that Twitter's basic design needs to be scrapped & recreated, a major headache. In tech, if you have to redesign 10%, it's cheaper (but painful) to redesign. New $15 million will be good for something.. - Mitchell Tsai
Hate to say it but I told you so on your blog the other week when you disagreed with Dare Obasanjo. Twitter given its current usage from its members is based on a flawed architecture. I don't think it can be fixed. - Jamie
Seems like VentureBeat is going for the hyperbolic headline. They know by rattling/linkbaiting Scoble they'll get some extra traction. With that said, it's not very smart singling out "power users" as the reason the backend is flawed. Would have been smarter to blame the system and leave it at that. Your power users are typically the most loyal. Don't blame them. Ever. - TDavid
Mitchell: That's exactly what Twitter has said it is doing. It's basically replacing its back end, bit by bit. - Ian Betteridge
Ian: The wording is awful. You don't say "it's the popular users!" You say "It's because we screwed up with our architecture" - Andrew Dobrow
Lynette: In entrepreneurship, I encourage people to create "Top 10 dreams" and "Top 10 risks" lists. A "Top 10" risk is all your founders die in an airplane crash. A "Top 10" dream is you're featured on the NY Times and 1,000,000 users show up in the next week. It's important to...at least...think about the possibilities and the costs/benefits of approaches to handle things. e.g. Your five servers were just burned in a fire, can you restore in 2 hrs? 24 hrs? Do you have emergency IT support? - Mitchell Tsai
Andrew: It's worth reading the whole of the architecture Q&A (http://dev.twitter.com/2008/05...). That puts this comment into a different context. Basically, they're saying "we screwed up" - but VentureBeat is grabbing out of context statements to get traffic. - Ian Betteridge
Andrew: That's the responsible approach! "We need to redesign our architecture for real scalability. We underestimated the demands of real users." Big companies like Exxon are not the only ones shirking responsibility! Little ones do it too. - Mitchell Tsai
Alex said it was a flawed architecture. He was asked what could be done right now to help alleviate (sp?) twitter's problem and he gave his answer based on the current state of the architecture.[Edit] I don't see the need to take offense at all - he did not say it was the "popular" users fault that twitter's so bottlenecked. He has already acknowledged in a previous post on twitter arch. that the arch. is at fault. If you want to take offence, go ahead but if you calm down and think about it, its a developer architect's assessment of what will help now. - Kamath ॐ
Oh and Scoble, I know you're trying to be quiet not to wake the baby, but please go record on Seesmic when you can talk at a regular volume. You sound like a cross between Freddy Krueger and Mr. Rogers. - TDavid
DAMN YOU, SCOBLE!!! - Ben Rhudy
It's obvious that once someone with lots of followers sends two thing in quick succession, he will generate a butterfly effect. But is it the person's fault (i.e. a bad thing instead of an achievement) that he is very popular? Notwithstanding, blaming users is the equivalent of shooting yourself in the head, company-wise. - Rodrigo Jaroszewski
TDavid: >>Freddy Krueger and Mr. Rogers that was the first thing that made me laugh in this whole thread, thanks! - Robert Scoble
Robert: Seesmic videos do not seem to open and run "inline" in IE7 -- a point I have made several times (trying to do it directly to Loic LeMeur -- with no fix and no response). Since you have a line of communication to him and I don't maybe you should pass the information on. - Brian Sullivan
While I didn't read any blame placed on Robert - If a single fellow can destroy such a service (which I think is complete utter crap) then you've got to examine your service. It isn't scalable. It isn't working. It's design was badly implemented, and it needs fixing. After all, what happens when 100,000 'scobles' come along? I think we should coin the word 'scoble' as a social network unit of peak activity :) "What happens to our services when we reach 1million scobles?" But I digress. - Mo Kargas
The simple answer, and the one Twitter should have used all along was "We screwed up." Period. Don't blame API calls. Don't blame power users. Don't blame Rails or anyone or thing else. We screwed up. We are sorry. We'll fix it. That's all. Anything else just adds to the current "Grab a torch and pitchfork" response from the community. I've given up that they will ever understand WHY it borks regularly, but maybe with the $15 million, they could hire a decent PR firm who can give them their talking points. - Cyndy
Wow, you would think that the admin od Twitter would be thanking Scoble, Laporte and Calacanus for making Twitter what it is today! If it wasnt for a half dozen heavy users, Twitter would have fallen into the bucket of useless crap on the cloud. if you (Twitter Admin) can't fess up to what the real problem is, maybe you need to hand over the baton and get out of the business. Aparently you don't like the spotlight and do not know what to do next. You are also saying that you have too many users and are literally pushing them out of Twitterland by the hundreds. I am on the fence and teetering to FriendFeed! - Pentaxfan
@ Cyndy, couldnt agree more! - Dudu P
Brian: I rarely use IE anymore. Firefox is just so much faster of a browser it isn't even funny. - Robert Scoble
Why all this talk about Twitter? Just forget about it and move to FriendFeed. - TranceMist
A sign of the beginning of the end is when they start pointing fingers at others. They see something else, like friendfeed catching up and soon will pass them if they don't get their act together. - Caleb Easterwood
Scoble - They said that "Not running scripts to follow thousands of users at a time would be a help" and it's something they have to address. How does this blame you? As a matter of fact, it's probably not even related to you but to somebody that was creating the db problems in the past week by mass amounts of follows. The article from VentureBeat is actually what's blaming you. :-\ - Damon Cortesi
Maybe, just maybe, the world isn't revolving around Scoble? nah! - Soulhuntre
via twhirl
Nice response Scoble - total bullshit that one person can take down a whole service. Look at some of the spam twitter accounts - I have seen a lot more follows on those than you have. What a crap post (VB). - Dave Gray
Twitter didn't blame Scoble, this guy Siegler who doesn't work at Twitter is the one who said it. To add -- if you look at the Q&A from Twitter, they were responding to a user who asked if the users could do anything to help. Twitter responded by saying don't run scripts that automatically hit their servers. Twitter did not say anything about Scoble. - Jason Honingford
So if the top users are part of the problem then folks who follow all the top users... wait, that means they're blaming me too. - Jon Winters
Well they could always kill themselves off by turning off the API and forcing people to only use the webpage again. - Jason Mitchell
via twhirl
I agree with Damon. "Not running scripts to follow thousands of users at a time would be a help" doesn't apply to Scoble, Calacanis, Laporte, etc. That's more of a spammer issue. So that could explain why there are problems when you're not on. Tools like this are a big problem >> http://www.livelybrowser.com/i... That tool looks at EVERYONE who is online and adds them as friends. - Mike Doeff
Twitter makes funny news now ;)) I still think you should be ashamed of yourself Mr Scoble! :D - Sébastien
Robert, you're not to blame for a service taking a dive. However, you've got it completely wrong when you compare FriendFeed to Twitter. The architectures are different and different goals. If FriendFeed had to analyze each person's SMS and IM settings for update notifications every time anybody posts *anything*, they'd be in the same exact boat. FriendFeed is not going down because they aren't pushing updates in real time. - Sol Young
I actually sympathize with Robert this time. Twitter *knew* letting someone push the system would hurt them, but unlike Facebook, they decided to leave things uncapped. And as soon as they reach a point where they think they can live without his free promotion, they make him the scapegoat for their scaling issues. - Roger Benningfield
Totally agree with you Scoble. Blamming the users is the wrong way to go. - David Cohn
Two things. a) The post that seems to have upset Robert was posted by MG Siegler on VentureBeat. That's the person that singled out Robert Scoble. It wasn't Twitter. On the Twitter Dev blog, they offered an explanation of where some of the large query volume is originating. That's NOT the same as BLAMING those sources for the current issues. From other posts it's very clear that they know their current architecture is inadequate. - Dewald Pretorius
http://xrl.us/bmazc tweets totally f#(%ed twitter - ⓃⓄⒶⒽ ⒹⒶⓋⒾⒹ ⓈⒾⓂⓄⓃ
Interesting points of view. I posted something, in spanish in my blog, about Twitter and community. http://robertoarancibia.cl - Roberto Arancibia
Personally, I blame Scoble for making me drop my ice cream cone. - CatCubed
