 |
|
| FriendFeed is the easiest way to share online. Learn more » |
david koblas › Likes
In-app purchases not so hot for some developers - http://www.tuaw.com/2009...
david koblas liked this
Not surprised. I really like the concept. Especially since it's a great way to sell extra "maps" or "cities" like Lonely Planet app instead of clogging the App Store with each individual city. I can see it as a good way to add "levels" to games as well, but I see the hesitation from users that already bought the game and now have to buy "add-on" content. It's very Sony PS3 Store-ish with their add-on system. I'm not sure how well that does either.
- John Wang
Facebook makes FriendFeed tech open-source - http://vator.tv/news...
Daniel Rowley, Cornelius Toole, Matt Cutts and 17 other people liked this
How should I feel? :-\ Is this good news or bad news?
- Olivia Lovag
I'd say good news :) Open source = more developers = more apps = more users.
- 'Like' robot (frɐnc)
Facebook has made a lot of it's technology open source (cassandra, hive, thrift)
- Deepak Singh
from iPhone
WHAT?? Wow
- Josh Haley
from iPhone
The Guide to the Freedom and Responsibility Culture at Netflix - http://www.slideshare.net/reed200...
david koblas liked this
Building collapse shows Turkishness - ABC 080209 - http://www.youtube.com/watch...
  |
sha-mayn teh, Brian Watson, Dave Winer and 6 other people liked this
Oops. The title of the video is a bit weird though.
- Turker Keskinpala
Btw, the video title is from YouTube, not my words. (I'm not sure what it means, but in case it's offensive...)
- Paul Buchheit
RT 10 UI Design Patterns You Should Be Paying Attention To | How-To | Smashing Magazine - http://www.smashingmagazine.com/2009... (via http://friendfeed.com/douglas...)
Bogdan Costea, david koblas and Mike Elliott liked this
This is awesome! If you treated everyone like your vendor: http://www.jasonmendelson.com/wp...
david koblas liked this
agile42 | Scrum Cheat Sheet - Scrum Overview - http://www.agile42.com/cms...
david koblas liked this
Nelson Minar on building social capital in multiplayer games - http://www.somebits.com/weblog...
Philipp Lenssen, david koblas and Amit Patel liked this
using an avatar creates a barrier to real-life interaction
- Andy Baio
No-one Reads Your Stupid Tweets t-shirt ($21) - http://rumplo.com/tees...
Darren Heydon, Didier Lahely, Louis Gray and 208 other people liked this
A+++... I tweeted this.
- Phil Essing
Want.
- Joel Webber
i need to wear this to remind me of my own insignificance
- Cee Bee
good. and maybe our g.o.v. also agree of this, so they just blocked the Twitter site.
- yagami
Pretty funny! And in most cases, pretty true.
- Colleen O'Donnell Pierce
I want a shirt that asks "if your followers are so loyal why don't you leave twitter?"
- Noah David Simon
Gotta tweet this :)
- Stephanie Segel
Two things strike me as amusing about this. 1) The fact that so many of us liked this entry (and, by extension, undoubtedly like the shirt). 2) The probability that many of us actually still use Twitter, conscious of the fact that practically no one really cares what we tweet :D
- Adam Lasnik
it could've been funnier if there was an another line of text => 'about 1 minute ago from txt in reply to @aplusk'
- Ozan Caglargil
Hi Jess, thank you for spreading the word about my cheeky t-shirt design ! If you'd like me to send you one just message me !
- Tweet Tees
WANT.
- John (a.k.a. dendroica)
+10 Ozan!!!
- guruvan (Rob Nelson)
from PeopleBrowsr
yes, finally somebody says it!
- Jacob
US Pole Dance Championship 2009 - http://www.youtube.com/watch...
 |
rob friedman, ǝuǝƃnǝ, Andrew Trinh and 12 other people liked this
The really crazy stuff starts at 2:00. There's no nudity, btw.
- Jess Lee
Wow, have to say that those girls get down!
- Shevonne
They should make this an Olympic sport
- Rodfather
That looks fun...a lot of strength and flexibility involved, like gymnastics.
- Anna Haro
Those chicks are um muscular - geesh
- BEX
The ex-roommate was given a pole as a birthday present last year. It's become quite the exercise activity, we can see why from this video.
- ♥patricia♥
The background music is all wrong.
- Richard Chen
Holy Cow - those gals are flexible!!
- Jannifer @wordsforliving
craaazy
- Yanting
rocketboom:
WTF Google?
Sometimes Google suggest works great, sometimes not so much. (via) - http://pop17.tumblr.com/post...
 |
david koblas, Alejandro, sam and 2 other people liked this
HAHAHAH
- mjc
Indispensable Link Building Tools - http://wiep.net/talk...
 |
Eric Logan, David Knight and david koblas liked this
"A good link building toolbox is invaluable. Some tools provide extra information or insights, while others point you to possible new link targets. Using a combination of various tools can without a doubt lead to analysis paralysis, but the right set of tools (and no more!) can put you in the right link building direction. In this post, I’m not going to name all link building tools, but simply the ones that I use."
- AJ Kohn
from Bookmarklet
RT @loic "We all need to create things we can be proud of. We need to make a difference" Richard Branson
david koblas liked this
Kevin Smith tells the hilarious story of his Superman script - http://www.youtube.com/watch...
 |
Steven Perez, Robert Felty, Sam Harmon and 17 other people liked this
These Jon Peters anecdotes are GOLD.
- Andrew C
This is a great story.
- Paul Buchheit
In Hollywood you fail upwards.
- Gabe
Giant spider...
- thepete
I laughed so hard at the end of this when he revealed the conclusion of the spider thing!! This vid is part of a brilliant series of speaking events that Kevin Smith did at various colleges. I think they were compiled into a DVD called "An Evening with Kevin Smith". You can watch the whole thing on YouTube and it is HILARIOUS: http://www.youtube.com/view_pl....
- Jess Lee
http://www.youtube.com/watch... - a bit of a followup.
- Andrew C
Shadow Physics gameplay demo - http://www.youtube.com/watch...
 |
Dave, Chris Prince, jug and 4 other people liked this
also: Miegakure, an experimental game in four dimensions [via]
- Andy Baio
Hmm... What's the date today? RT @chrisfralic http://www.smellr.com/
Brendan Jackson and david koblas liked this
How FriendFeed uses MySQL to store schema-less data - http://bret.appspot.com/entry...
Francesco Marconi, Jorge Escobar, William Kelley and 214 other people liked this
This is a technical overview of the new backend system we launched a couple of days ago.
- Bret Taylor
First question, why MySQL instead of a straight key/value/attribute store, like a bdb? (Amazon's SimpleDB works along similar principles, btw.)
- DeWitt Clinton
Cool stuff Bret. Love to see this kind of stuff. Would love that chat about the Ajax side of things ;)
- Dion Almaer
DeWitt: Historical reasons; we have some operational experience maintaining MySQL servers, and MySQL is very popular so there's a lot of support available.
- Tudor Bosman
Oh right, none of you are ex-Amazon. (Any Amazonian reading this will understand...)
- DeWitt Clinton
How small do you keep the shards? Can you keep the entire index tables in RAM? Are you backed with SSD?
- DeWitt Clinton
Not all of it is RAM. We use normal hard drives for this system, and we have enough RAM such that the working set fits in memory. Most of the oldest stuff on FriendFeed is not accessed frequently, and that data is not typically resident in memory.
- Bret Taylor
Thanks Bret, nice post.
- Ömer Faruk Kurt
So for something like 'user_id_index.get_all' you'd trust that InnoDB has that user in RAM, and that you have only a small set of active users at any given time (for web requests). Same for the relevant entities you'd "join" against. Makes sense. (Sorry, having a hard time typing coherently tonight.)
- DeWitt Clinton
Yah, those indexes are typically in RAM. Our indexes contain a timestamp as well (since they are ordered reverse chronologically), so the oldest entries referenced by that index would likely not be in RAM depending on how many pages back they are on most feeds.
- Bret Taylor
BTW, did you guys look at any other serialization mechanism other than pickle? Any pros/cons for cpickle?
- Arvind Sundararajan
cPickle is the same format as pickle, just implemented in C instead of Python. We use cPickle. marshal is faster, but according to docs may change from version to version of Python, so we didn't want to use it in our DB.
- Bret Taylor
@Bret "250 million entries", is it possible to give the size of the DB? and what is your opinion about when will you need to change this system again or is it strong enough for years Meybe you will only need hardware instead of structure changes?
- Ömer Faruk Kurt
Ömer: I don't have an exact number off the top of my head, but a lot of data is the indexes themselves, so adding indexes is almost as significant as adding entries.
- Bret Taylor
This sounds exceptionally similar to how the AppEngine datastore does things, except that it's MySQL and not BigTable.
- Alex Power
Aren't commercial RDBMSes (like Oracle, MSSQL) designed to support the sorts of things you're looking for, like online reindexing and joining across shards? Why not use them? Lack of experience? NIH? Too expensive?
- Gabe
"We like MySQL for storage, just not RDBMS usage patterns" Excellent post Bret, thanks for sharing!
- Mahesh CR
Gabe: "designed to support" and actually working are two different things. I think that is why companies like ours generally don't use commercial software and choose to use open source software. If it doesn't work, we can fix it. In my experience, most of those features don't work at the scale consumer web sites need, and it certainly wasn't worth the cost and time to find out from our perspective.
- Bret Taylor
Non-RDBMS-patterns may work when there is no great needs for processing the actual data (like reports etc). In LOB-apps this would be a big no-no, but in large-scale services like FF alternative methods are almost necessary...
- Jemm
Thanks for the writeup! Is there a reason you don't declare the added_id as an UNSIGNED INT?
- Roger
Roger: no, no reason.
- Bret Taylor
@Bret, this is a pretty cool write u!. Might I suggest as a subject for future write ups, how the realtime update system works (e.g. how it knows what updates it needs to push), and perhaps how the feed fetching system works and how these might inter-relate?
- Ray Cromwell
Ray: good ideas. We will do more of these. We want to in theory - they just take a bit of time to write up, and we like writing code more than blog posts :)
- Bret Taylor
Bret, do you really think that guys like Oracle and MS create features for enterprise-level databases that don't work or scale to meet the needs of those who buy them? Of course you may be right, but the TPC tests generally get results in the hundreds and thousands of transactions per second. In reality, though, what can you really fix if something doesn't work? Are you going to debug MySQL if it starts corrupting data or optimize it if it's too slow?
- Gabe
Gabe: yes, we have already debugged MySQL when it has crashed. It would have been impossible without the debug symbols and source code.
- Bret Taylor
Gabe: I've been doing a fair amount of tracing through MySQL core dumps over the past week, and just having the ability to look at the code, plus the size of the community working with the same code base and debugging problems, seems to make it worth the cost at this point. We haven't changed any of the MySQL code yet, but Google and others certainly have.
- Jim Norris
This is the best thing I read today.
- pastas9
I'm skeptical of Oracle and other commercial pre-packaged systems because the companies are focused on extracting revenue via software sales and their products are highly optimized for a certain problem domain that doesn't really fit our experience very well.
- Jim Norris
So I guess the downside is that if you change your indexes you need to change the code, too. How do you manage the table creation, code migration and data sharding all at once?
- Nick Lothian
Nick: It is actually not bad: make the tables, update the code, start writing to the indexes for new entities, then run the "Cleaner" to fill in the indexes for the older entities. When the cleaner is done, you can start using the indexes for features.
- Bret Taylor
The most important decision about your design- and why not going with databases RDBMS etc. to manage the indices - is that your users don't really look up the old stuff, so keeping the old entries in RAM/live indices isn't worth the time hit it would take to do huge indexes on long tables. This seems like a "running index"- that is, it's populated by the application in many places at one time, instead of in one place, that the DB then optimizes over the entire history.
- anna sauce
Reminds me of some multi-user java apps back 10 years or so ago that had to manage simultaneous users on live systems. great post, Bret, and fun to see what's working behind the scenes.
- anna sauce
"if you change your indexes you need to change the code" is not a problem unique to this system. You always have to update your code along with your schema. You have to write code to read from the tables even if you use a typical RDBMS, and you can't do that without an index, so this staged process exists in some form no matter what your storage scheme looks like.
- Bret Taylor
Bret, it sounds like MySQL crashes a lot. Are you assuming that commercial products will crash just as often and the vendors won't debug it?
- Gabe
Gabe: Every piece of software crashes, whether it is written by Oracle or by open source developers. The difference with open source is that we are not dependent on someone else to diagnose and fix the problems. This is not a unique sentiment. Google, Yahoo, Facebook, FriendFeed, Twitter, et al, have all chosen open source infrastructure for this and a variety of other compelling reasons we have already discussed.
- Bret Taylor
Brett, I agree that not being dependent on somebody else is great. In fact, the product I work on is probably 99% open source or developed in-house. I even replaced a relational DB with a pickled Python object store a year or two ago. However, I'm also not considering writing my own transaction protocol, indexing, and query engine because the open source DB I use doesn't support...
more...
- Gabe
Re: "change your indexes you need to change the code" - yes, conventionally you need to change your code if you change your tables. But you do get some opportunities for runtime optimisations by changing your indexes (although in practice this can have availability costs as you note)
- Nick Lothian
OK I'm going to put this out there: sounds like you're re-inventing the wheel.
- anna sauce
Being able to open something up to understand, diagnose, and fix things yourself is underrated.
- Amit Patel
DeWitt, part of the reason we didn't use a simpler store is that we're using mysql replication. We'd have to replace that too. Probably not hard, but it's working well for us. Also sometimes you do want transactions for performance within a single DB (each xsaction is a single log write, vs multiple little writes).
- Private Sanjeev
re: SSDs and mysql. I can't speak for all SSDs, but the Intel X25-M gets internally fragmented very quickly, reducing write performance by 10-20x. Things that do large sequential writes like bigtable or lucene are a better fit.
- Private Sanjeev
Part of the purpose behind traditional schema driven designs is to 'protect' the data from the programmers and the applications - which is not insane given the level of commitment people have in some kinds of organization. One of the statements that the FF people are making here is that they have trust in themselves and each other and they care about what they are doing.
- Robin Barooah
You guys rock so hard it HURTS! THANK YOU, FRIENDFEED!
- Josh Haley
Just logged in... this sounds really cool.... hmmm :)
- Susan Beebe
great post -- liked the discussion in the comments too! some people thinking relational databases can do everything; other people not understanding rationale behind key/value store; other people mad you've reinvented it using MySQL... :-D favorite comment from guy who is having a heart attack because anyone might think it's a good idea: "it is like using a database to solve the problem...
more...
- Karim
what is the correct name for this? "schema-less data?" "entity attribute/value (EAV) store?" "key/value store?" personally, i like "property bag" :-D also, have you considered Amazon SimpleDB, or Microsoft SQL Data Services for hosted services, since they are based on similar ideas, just not on MySQL?
- Karim
Sanjeev, I heard rumors about the X25-M performance running down after fragmentation, but last I heard was Intel was unable to replicate & "looking into it." 10-20x should not be hard to spot. :-) Are you in touch with Intel...? Also, have you seen similar problems on SSDs with Samsung controllers? (I went with the Corsair...)
- Karim
Karim, it may not get as bad with other SSDs, but they start off with much higher write latency for small random writes than the X25-M.
- Private Sanjeev
The Samsung, you mean? All I was trying to do was *not* get the JMicron controller and *afford* it :-D
- Karim
but why are the ssIDs varchars, is the question (from @eonarts comment)
- anna sauce
anna: what are ssIDs? If you mean the UUIDs, they are just 16 byte binary strings.
- Bret Taylor
Just look how many comments on this begin with someone's name. "anna:" "Karim," "Sanjeev,". Somehow there needs to be a reply-to-comment mechanism. Fortunately FriendFeed uses json blobs in their database so this won't require a painful schema change ;-)
- Kevin
Bret: you're right, I re-read the post and couldn't find evidence of the Primary Key Erin mentions in her comment, as varchar. Am I missing it?
- anna sauce
schema changes aren't painful, and I have a question for more current DB/programmatic people (than me): why is this called schema-less? There is a schema, it's just that the database application isn't being used to index, Bret's crew are doing it manually. Blobs in columns doesn't mean it's schema-free, that's been going on for 15 years. I think it's just that he's cracking the indexing methodology and offroading. I'd like to hear from some PostGres people on this- Disqus is on PostGres aren't they?
- anna sauce
Anna: that person is talking about the URL index. What we are doing is correct for this application; we want the index to be in that order on disk, not in a different order.
- Bret Taylor
Bret: ah, ok
- anna sauce
I just remembered that one of my client's DBs is actually Btrieve, which is nothing more than a way of storing binary blobs with certain byte ranges indexed. (They are still using a DOS app from the '80s, with the Btrieve running on a Novell server.) The only real difference is that they have fixed-length keys and records, while the MySQL data is variable-length.
- Gabe
Bret: Thanks a lot for the informative post! Any thoughts on MySQL vs Postgres? We found tables crashed with too many simultaneous writers in MySQL, but your post says it "doesn't corrupt data", so I guess you haven't had such issues.
- Ruchira S. Datta
Bret: Do you apply the same approach to perform searches on your stored text data? (i.e. inside the data storing the BLOB)
- Roger
Bret, do you care to explain briefly how you approach the following/follower aspect without JOINs? I loved the article, keep reading it every once in a while...
- Jorge Escobar
Goat jumping over notch at Comeau Pass.JPG (image) - http://lh4.ggpht.com/_3KVrWN...
Gary Burd, david koblas and Private Sanjeev liked this
Prepare yourself to laugh; then click on this http://www.altgate.com/blog...
david koblas liked this
Infrastructure for Modern Web Sites « random($foo) - http://randomfoo.net/2009...
pastas9, Kevin, Joel Webber and 17 other people liked this
"One of the things that I did when I wrapping up at Yahoo! was to begin to take a look at the current state of web frameworks. I ended up picking Django, but I have to say, I was disappointed with the state of what’s out there. Friends will have heard me bemoaning this sad state of affairs - that while Rails and Django might make CRUD easier, that the ORMs weren’t suitable for scaling beyond “toy” sizes, and that more importantly, they didn’t seem to address almost any of the pain points of building and maintaining a modern website."
- Bret Taylor
from Bookmarklet
Java has some nice ORMs. I've found that Guice + Hbernate/JPA gets you ORM scalability with minimum code, without the Spring monster. But I'm not willing to go the JSP/Seam/Tapestry/Wicket route, too much junk. I think marrying REST JSON/ProtocolBuffers/RPC with a GWT front end and hibernate backend solves alot of the issues surrounding server-side template frameworks, and their attempts to hide HTTP stateless-ness behind abstractions like continuations or in-memory UI widget trees.
- Ray Cromwell
And if you're willing to get your feet wet with Scala, it has a nice activerecord-ish ORM and web framework called Lift, but I still prefer the REST approach.
- Ray Cromwell
ORMs are very low on my personal list for infrastructure that I want/use to develop web sites. Deployment tools, A/B testing infrastructure, etc, get much less attention than they deserve relative to things like ORM.
- Bret Taylor
I just find it hard to trust ORM, I have in the past been burnt by systems that made it incredibly hard to deploy changes - and I have typically been involved in systems where database optimisation mattered, and the choice of presentation/templating layer was key to productivity, as did reliability, easy change and deployment.
- Joelle Nebbe (iphigenie)
Bret, I'm with you on not really caring for ORMs. But what about the rest of the FriendFeed implementors? I get the impression you all are using something to abstract away database calls.
- Bruce Lewis
from fftogo
It seems like a lot of hit below-the-line stuff can be handled by services like App Engine. But then he runs into the potential issue of his above-the-line stuff not working as the kills most Django apps that would address those issues.
- Brett Cannon
Very lucky kid on The Price is Right - http://break.com/index...
Karl Rosaen and david koblas liked this
How to Discover New Content - http://www.louisgray.com/live...
Lu Tao, Niki Costantini, Anne Bouey and 32 other people liked this
Post by Michael Fruchter.
- Louis Gray
Nice post. Would love to check out toluu. Can someone pass me an invite?
- Vishwas Narendra
Anybody who wants Toluu invites, send me an e-mail to louisgray@mac.com.
- Louis Gray
Nice post Mike. Good overview of the current landscape.
- Caleb Elston
Yay, me in a screenshot! Thanks Mike :)
- Phil G
nice post, thanks to the intro to toluu, anyone have invite? me@seanpercival.com thx
- sean percival
Sean, it is on its way.
- Louis Gray
I like. I sent you an email too Louis. Thanks.
- Amani
Thanks. Now I need to build my friends list on toluu. I wonder how. Is it tied into FF anyway?
- Amani
Amani, We are working on some friend discovery tools. Twitter, Email, and possibly FF too.
- Caleb Elston
User Interface Concept: Calculator « petitinvention - http://petitinvention.wordpress.com/2008...
Shakeel Mahate, Kelly Norton, Peter Fedin and 17 other people liked this
This guy's blog is fun - constantly posts ideas for products he has. Always interesting.
- Bret Taylor
from Bookmarklet
this is amazing stuff!
- Jorge Escobar
So could you literally carry the one then?
- Louis Gray
Human Slinky Is Freaky - http://xorsyst.com/videos...
Richard Chen, Nick Lothian, Rachel Lea Fox and 8 other people liked this
Amazing.
- Paul Buchheit
Awesome
- Jim
Yes, very interesting... I wonder what it feels like to be inside that costume? :)
- Danielle Closs
The Dark Knight Trailer Recut - Toy Story 2 - YouTube - http://www.youtube.com/watch...
 |
fshamsi, Ross Miller, david koblas and 2 other people liked this
The best moment is when Michael Caine shows up as the voice of a green plastic dinosaur
- j1m
Brilliant!
- Christopher Chung
FriendFeed birthday cake.
Emily, Dan Hsiao, Brian Johns and 19 other people liked this
WOW!
- zizukabi
Is your son's birthday (I assume the child in the previous picture was your son) on the same day as FriendFeed's?
- Sebastian Kanthak
Sebastian, it's a picture of me on my first birthday.
- Gary Burd
Haha, and I thought you liked these old wallpapers so much that you put them in your house. :)
- Sebastian Kanthak
Other ways to read this feed:
