"Mojo is music sharing done right. With just two clicks, you are ready to browse, select, and download music from other Mojo users. Plus, subscribe to the playlists of other users, and Mojo will update those playlists anytime they are online. Finally, all downloaded songs will automatically be added to your iTunes library."
- Matt Frog
from Bookmarklet
If you sign up and install, please do share your username if you'd like to connect! (Abba fans need not worry themselves ;-) ). I am mattuk@deusty.com
- Matt Frog
Just set this up, after hoping I could get some of my mates to install it. neurario@deusty.com
- Neurario
Yes, all this hosting and file serving is free to provide anyway. They must just be trying to really screw us by raking in even MORE money with those ads. How evil of them.
- David Chartier
from iPhone
skipscreen was working before but it stopped working for rapidshare recently. If you are downloading lot of stuff it makes sense to actually pay for rapidshare month by month, with no yearly commitments, mind you.
- TrafficBug
Nmap 5.00 Released July 16, 2009 -- Insecure.Org is pleased to announce the immediate, free availability of the Nmap Security Scanner version 5.00 from http://nmap.org/. This is the first stable release since 4.76 (last September), and the first major release since the 4.50 release in 2007. Dozens of development releases led up to this. Considering all the changes, we consider this the most important Nmap release since 1997, and we recommend that all current users upgrade. About Nmap Nmap ("Network Mapper") is a free and open source (license) utility for network exploration or security auditing. Many systems and network administrators also find it useful for tasks such as network inventory, managing service upgrade schedules, and monitoring host or service uptime. Nmap uses raw IP packets in novel ways to determine what hosts are available on the network, what services (application name and version) those hosts are offering, what operating systems (and OS versions) they are running,...
- Bwana ☠
When it comes to programming techniques, screen-scraping can be a complicated and annoying thing to deal with. I’ve dealt with it quite a bit in various projects, most recently my TV Library plugin for boxee. I use PHP to accomplish most of my screen scraping and its got a pretty great arsenal. However, if ease [...]
- Jake Marsh
This being the only way to post an entry, a picture AND the first comment at the same time, I propose that the bookmarklet, when invoked over a FF page, automatically links to the topmost story on that page [here: http://ff.im/4O5GX], rather than to the feed, or search result itself. How's that for a simple enhancement?
- ianf ⌘
from Bookmarklet
I would rather enable comments in the "share" pop-up.
- Jérôme
That's a parallel possibility. Another would be to have the bookmarklet stub code return a dialog with a different "Post New" button, or an additional checkbox "Unlink this new post" to decouple it from underlying base page.
- ianf ⌘
I dunno, I've been using Launchbar (the beta) for a while and I prefer it.
- Jalada
i dont mean to sound aggressive Jalada, but I'll put my house on it that Quicksilver can do everything and more better than Launchbar
- Zee.
Oh yea zee. Qs can do a lot more than Launchbar. I do prefer Launchbar anyway :)
- team bonino
I couldn't use QS for more than search & running apps. And for that the Spotlight works well, which made me remove QS. BUT, knowing it now posts to friendfeed directly, I guess I'll try it again...
- arnaldostream
After reading this review I guess I should install this as the next level of **Pimping my Mac**
- Jonathan.Rivera
I'm a long time fan of QS, but -for the sake of infosharing- you can do quick unit conversions using the Google Quick Search Box as well. Simply double-tap the command key and when the search box comes up type, for instance: 100 miles into km. - The first choice offered should be "Search Google..." - Hit that, and Google will do the unit conversion in your browser.
- Carlos Granier-Phelps
I'm sure Quicksilver can do it all, but I always found Quicksilver's configuration to be cumbersome and complicated. Maybe I was just doing it wrong :P
- Jalada
I used QS for a long time and love it but found it ate a lot of resources on my old powerbook so I stopped using it. I tried Butler and although very different metaphor it did everything QS did but again found the same thing with resources. Now I use the google quick search which is nowhere as hungry but doesn't really cut it yet. As I'm doing more and more in firefox now I'm using ubiquity more and more and that is rocking.
- Brett Jackson
Shedali: as far as I know, there is still some active development ongoing by members of the quicksilver community. However, Alcor, the creator and lead developer of Quicksilver, is now heading up a team at Google developing the Google Quick Search Box: http://code.google.com/p...
- Brandon Titus
I would love to have a Quicksilver guru help me use Q more effectively. Seem like I'm mostly using it as an app launcher. Any tips INS goodsite for this?
- JR Holmes
from Nambu
I was a satisfied user of Quicksilver until OS X 10.5 came out. Quicksilver became unstable on it and some plugins didn't work anymore. I now use Google Quick Search Box which looks more stable and promising than QS.
- Brome
I dumped it last year - Quicksliver is too complicated and it's balky on Leopard.
- Bill Sodeman
I just bounced back over to Quicksilver after an extended relationship with Butler. Butler's interface seems a little more simple to me, but QS is fast, super extensible, and I like the QS plugins.
- Joel Zehring
I can't live without QS and will be heartbroken if it stops being compatible with the upcoming os releases.
- wynlim
I can't live without Quicksilver, first thing I install on a Mac
- Simon Tracey
My problem with Quicksilver is that I become SO dependent on it on my own mac that I can't seem to do anything when I get on a Mac that doesn't have it.
- Jeff Smith
from twhirl
Recently I haven't been using it that much, but that's because I've been spending less time at the computer.
- Richard A.
Quicksilver was indeed ahead of it's time ESP for those us us who prefered not to use mice - before spotlight using the GUI on a Mac wasn't my favorite
- thinfilms
from Nambu
I wish I had a nickel for every second of time QS has saved me...It's a must have.
- Chris Rossini
geek search? hm... what about searching for drivers for your ubuntu installation or linux how-to's and guides. uber geeknes ftw. =)
- Apostolos Papadopoulos
Apostolos: no, I'm talking about searches for Twitter Search http://search.twitter.com or friendfeed search, like I'm posting here. One thing, if they are too noisy I just make them display things that have a "like."
- Robert Scoble
Spam filter for all searches, use service:internal, http://friendfeed.com/search... (because there are many unnecessary facebook, ping.fm, blog entries)
- Alp
Robert: I'm a nerd and I really like friendfeed too. Twazzup might integrate with friendfeed in the future, but right now most of the real-time-web content is on twitter. If there are things we can do with twazzup to make it easier to filter, I would love to hear your thoughts.
- Cyril Moutran
Cyril: I'm using Twazzup too. You're onto something. The money is in search, I hope you succeed! Where are you located, by the way?
- Robert Scoble
I really like Twazzup more then Twitter search. New entries come in real time.No need to refresh the page. Slowly the web is moving into real time updates.
- Keven
ruby on rails, ajax, palm pre, cloud computing, the cloud
- Angus Burton
what do rails, django piston, web2py and the Twitter and Facebook APIs have in common? REST. it's a style of programming described in a dissertation by http://twitter.com/fielding and it's something i'm always looking to learn more about
- Brian Hendrickson
thanks Robert! thanks Keven! - we're located downtown SF
- Cyril Moutran
I tried it when it was first released and it was too buggy and slow for me to use as a quicksilver replacement. I'll have to take another look at it.
- Phil Maxwell
It even let's you send updates to Twitter: "To tweet using QSB first set up your Twitter account (as documented above). Once that is setup you are ready to tweet. To actually tweet, you need to know the secret of creating a text item in QSB. Text items are things that start with a space (i.e. hit the space bar and then start typing). So to tweet you must activate the QSB, hit the space...
more...
- Peter van Teeseling
same guy developed it, if i got ir correctly
- Naor Mark
There's a Mac version of Desktop too. I've used both versions. QSB is quite a bit more useful.
- Jason Wehmhoener
I'm going to have to give this another look. It looks like they've made some more improvements. @Naor yes, Alcor, the developer of Quicksilver is the lead dev at Google on this project now.
- Adam Turetzky
I've been using it since the first release. It's replaced Quicksilver for me although there are still some things that I miss. It's quite stable.
- Brandon Titus
same here, it's a fantastic app I can't live without now.
- Jim Carter
@Adarm, that's a good reason to follow this one, should be interesting to see how it'll eveolve, it has to go some way before replacing QS, although it does feel great, wonder if they'll let plug-ins to be developed
- Naor Mark
Yeah, I'm a OneNote -> Evernote convert. Doesn't seem like we really have Window client parity with Mac yet.
- Ken Sheppardson
I use Evernote all the time to keep track of things, love it
- Kim Landwehr
Question for Phil: When will us Windows users have all the same functionality Mac users have? The "Notes by title" thumbnail view comes to mind. Seems to be in the online and mac versions, not on Windows.
- Ken Sheppardson
Are they developing a Palm Pre, webOS client?
- Ken Sheppardson
Would like to see plain old Jabber/XMPP/Google Talk integration, so I could just IM stuff into a notebook
- Ken Sheppardson
I love EverNote. I use it across Mac, PC and iPhone.
- Michael
just got my cigar aficionado magazine nice review of the palm pre don't know how they got one but great review
- Gary Prechtel
I just use the browser plugin rather than the software installed version
- Greg
I'm sorta confused whether Phil was making a Palm Pre joke or a dis...
- Ken Sheppardson
Evernote rocks! I wish I worked for them. They really have something to be proud of there. Ken: I think Phil was being coy because he couldn't talk about what they're doing with Evernote on the Pre yet.
- Joe
Knew about EverNote for a while but am just trying it now and LOVE IT, its going to rock during school with long science class's and lots of writing on whiteboards!
- Dakota O'Neill
OK, I think one of the terms of the Palm Pre early access dev program is you can't say you're part of the early access dev program... so now I feel better :-)
- Ken Sheppardson
Sorry if this was asked before..will there be Blackberry Bold support?
- Jeff Gignac
@Jeff Gignac Like he said they are working on a lot more phones. But you can always just use the email way. Which is just as good!
- Dakota O'Neill
what's the discount code for squarespace?
- Karma Martell
Leo, I thought we were doing a Gillmor Gang tonight. Is that still on?
- Robert Scoble
Yes, he said he is just about to do it!
- Dakota O'Neill
leo, in Skype video, right click on the video and click turn off return video or something to take off the PIP in Skype-Please
- Andrew Martin
If you can tell me how to turn off the PIP without turning off video for our callers I will. Bugs me too, but our hosts want to see the same thing you're seeing.
- Leo Laporte
Absolutely can't wait. Tweetie for the iPhone has everything you'd ever want in a Twitter client, IMHO. While Tweetdeck has a load of features, it's not Tweetie. Looking forward to this and will happily drop some coin for the app.
- Anthony Baker
Easy choice, Nambu, free across platforms :-)
- Richard A.
I like the Seesmic client very much but a real application not Air based is much more interesting.
- Donald Townsend
Looks good from the video. I'm guessing it'll be a paid app, just like the iPhone version.
- Dom Barnes
What is it with people and free? I understand that the economy is in the shitter (boy do I ever) but if you can't afford a $15 app maybe Twitter isn't the best way to be spending your time. Tweetie is developed by one guy just trying to make a living. Sheesh.
- jakebf
I'm building my own server in the cloud, thanks to Dave Winer: http://www.scripting.com/stories... but right now that will have to wait cause Maryam wants me to come downstairs and stop playing on my computer.
I'm not sure I'd recommend EC2 for beginners, even with a HOWTO. There are much cheaper cloud-based alternatives (like Rackspace's Slicehost offering). "Turning it off when you're not using it" isn't a solution. This isn't 1970.
- Mark Trapp
Jesse: no, for now I'm just following along Dave's podcast.
- Robert Scoble
After I get it running on Amazon I'll get it running on Mosso. Gotta run, Maryam's getting mad I'm still playing on the computer.
- Robert Scoble
Is s3 mounted as a filesystem on EC2 much faster than s3fs elsewhere? I'm doing rsync with s3fs on a linode server and it's taking all weekend.
- Bruce Lewis
Bruce: how big is your diff? That doesn't seem right.
- Mark Trapp
Amazon has Windows servers... which Dave uses in his example. Mosso Cloud Servers and Slicehost look like they are Linux-only for now. EDIT: GoGrid has Windows servers as well, which are even easier to bring up than they are at Amazon.
- Karim
I know there is a project that can use EC2 and S3 rattling around my brain, I just haven't been able to find it in there yet.
- Mike Flynn
I've fallen in love with all of amazon web services... hard. I have trouble imaging building scalable projects without it now. My current project uses ec2, s3, cloudfront, elastic ips, elastic block storage, simple queue service and simple db.
- Ted Roden
Karim: Mosso offers a Windows server (http://www.mosso.com/cloud.jsp) but it's a bit more expensive than leaving an EC2 instance running 24/7. It's hell of a lot easier to set up, though.
- Mark Trapp
Great idea Robert. Once you try those, I'm very interested on what I'd get from RackSpace that I wouldn't get from Amazon, or more importantly, what of all those you try out you think is best. I keep coming back to Amazon due to price and flexibility, but I'm always open to new things if they make sense.
- Jesse Stay
Mark, I thought those Mosso Windows offerings were Cloud *Sites* not servers -- i.e. basically a webroot where you get to upload stuff? From what I read, the advantage is that they automagically scale, but since it's not a server of your own, you can't log in as Administrator & install stuff.
- Karim
Ted, what do you do for load balancing on EC2?
- Karim
Karim: and what, exactly, is a beginner non-developer/non-IT/non-sysadmin going to be doing that requires a full server? I don't think people who require full control over the inner workings of a server are the people being targeted, here.
- Mark Trapp
Karim: On http://enjoysthin.gs I'm running one web front end so I don't need a proper load balancer. I split up some of the backend work to separate servers... same with the DB. But the app handles all that. In the past I've used both HAProxy and simple round robin dns.
- Ted Roden
Mark, let me ask Dave Winer why poets need servers and get back to you. lol
- Karim
Ted, thanks. Roll-your-own HAProxy seems to be pretty popular with the EC2 set. Amazon has threatened to come out with a load balacing solution at some point...
- Karim
Karim: they will. Don't get to comfy with any load balancing solution, amazon will come up with something that beats it.
- Ted Roden
Do what 'the boss' says ... or you're doomed (and she's right :)
- Charlie Anzman
I get that a lot from Amanda. Good to know I'm not alone:-)
- Brandon Mendelson
Mark Trapp - Sorry I fell asleep. The diff is less than 10GB. It looks like it might be almost finished now.
- Bruce Lewis
Read the comments. It's a pricy proposition, the more you use it. There are cheaper solutions, no?
- Ernie Oporto
from Nambu
I've been playing with EC2 for a few months now, including creating my own linux images that weren't (aren't?) included "out of the box". This also gives you the legroom when you're going to mount your S3 buckets for your own files. It's a simple and painless task if you know your way around linux/unix systems...
- Holger Eilhard
RackSpace has more instance types available than EC2. The lower end ones have less memory so they can end up being a lot cheaper. It may be worth checking out depending on your usage.
- Todd Hoff
"Attempting to inject some fresh thinking into podcasting is AudioBoo, a new service from the team at Best Before. Currently restricted to the iPhone, their mobile app allows users to record instant podcasts on the go. Forget having to deal with post-production tasks like editing and compression, AudioBoo podcasts (or ‘Boos’ as they are called) are rough-and-ready, spur-of-the-moment recordings."
- Martin Bryant
from Bookmarklet
The Gillmor Gang - Jason Calacanis, Marc Canter, Leo Laporte, Dan Farber, Robert Scoble, Andrew Keen, Loren Feldman, and Doc Searls. Recorded Saturday, February 28, 2009.
- Steve Gillmor
MC was baiting, but he and the others, including JC, knew it was driving the show forward. My read was that the back and forth never lost its equilibrium. The gang dynamic was in full swing, and the venting well represents pent up feelings in the wider social/economic climate. Good stuff.
- Micah
Great Gang... I think Leo's influence + Gillmor Gang madness = almost tolerable show. :-)
- Jason Calacanis
@stevegillmor RssPlayer is the podcatcher for the iPhone. It's so much beter than iTunes at this point. And you can email feedback to the developer. It's only getting better.
- Christian Burns
I agree with Jason. I have now started listening and enjoying this show since Leo joined. It's also nice that he manages everyone getting mic'd properly unlike some other shows where it was painful to listen to Steve do it :)
- Mark Krynsky
Does Leo dominate the show like on his TWIT podcasts? Or is it more of a shared effort?
- ComicList
I don't think Leo dominates HIS podcasts. I love hearing Leo's voice. And I haven't heard Gillmor Gang yet, but from what I saw from the Live stream, it was an interesting show.
- Molly Song ;)
I enjoy Leo Laporte's participation on GG. HIs followers? Not so much. The rudeness and self-indulgent flaming in those chat rooms is nearly intolerable.
- Karoli
Now that YTpodcaster is dead, RSShandler seems to be an OK alternative. Not as userfriendly as YTpodcaster, though. I'm sure it will die soon too, due to bandwidth issues - so is there any sort of open-source (or even commercial) alternative?
- Ingo
Sounds like some online video company should acquire ytpodcaster. If funding is the issue surely Boxee/Hulu/Youtube would be interested.
- Keith - @tsudo
"sitetweet allows webmasters to post immediate messages on Twitter about specific behaviors on their site. Usually installed on strategic pages, you can configure sitetweet to send a special message when a user performs a certain action on your site. "
- Zee.
from Bookmarklet
We just launched a new backend system for FriendFeed that should make things a lot faster. Let us know if you see any problems or unusual changes in your feed. Thanks!
is it like the backend of the new mac mini -- 80 USB ports?
- MG Siegler
Increasing the number of USB ports was vital to this upgrade :)
- Bret Taylor
Nice... you were doing what I was doing today... see you can make major upgrades WITHOUT down time. Nice Job FF!
- Brian Roy
Ok! Being behind a proxy makes my way somehow difficult but not that bad, I'll comment on that with stats if I see changes. Thanks Bret!
- Zu from AOD
I will write something up at some point in the near term. Maybe after it has been running for a couple weeks so we know it works like we think it will :)
- Bret Taylor
Hopefully this helps with the lag time that I'm seeing from Twitter. I like using FF as a Twitter client; it's been tough lately with tweets taking over a half hour to get here. Nonetheless, well-done, guys. :)
- Jared Smith
Sweet bejeesus, you guys rock! I like new features without taking down the whole site. This is efficiency baby.
- Scott Jarkoff
Zee: loading your home feed, especially if you have many subscriptions, will be a lot faster. In general, viewing feeds with lots of people in them is much faster.
- Bret Taylor
Bret, what about the RSS machinery? Will it fetch more feeds more often?
- directeur
That is somewhat independent of this change, directeur. However, it should improve feed publishing a bit over the course of the next fews weeks as we completely decommission the old system - it will improve some aspects of our publishing throughput. However, most of the feed fetching improvements are coming as sites like BrightKite and Identi.ca adopt SUP (http://simpleupdateprotocol.go...)
- Bret Taylor
one of the reasons I use ff so much is that I can often open a new tab, load friend feed, and read a couple of entries in the time it takes for some other site to load; nice to see that speed remains a priority :-)
- Karl Rosaen
Thanks for the details Bret, I like the SUP concept and I actually use it on my Jazz blog and on SocialWhois too :)
- directeur
Bret great to hear:) Keep up the great work.
- Rob Cairns
Fast is better than slow, after all :)
- Joel Webber
Bret - good stuff. Any effect on the delayed Twitter feeds? Or is that out of FriendFeed's hands?
- Hutch Carpenter
Yeah, we need way more USB ports! LOL :)
- Susan Beebe
It was quite fast to start off with, didn't notice any big differences. (Good nor bad)
- Umit Namli
2Bret: According to Alexa traffic graph, FF doubled its traffic in this week. Is it real and if its real what was the reason for that? If its not confidential, May be you can share with us :)
- Ömer Faruk Kurt
Ömer: Alexa is like a Ford T, it's a bit old. Don't know what to make of that service.
- Umit Namli
I still seem to get the FF is unavailable screen quite a bit. I'd say about 50% of the time when trying to do searches, filter by service (especially FF itself) or look at best of day. Usually after getting the unavailable screen if I refresh one to five more time eventually the page loads. Annoying though.
- Thomas Hawk
Thomas: yah, your account has been a stress test for our system for a variety of reasons. We hope to make progress on the Best of Day and search ASAP, and I will let you know. Sorry for the trouble.
- Bret Taylor
Does anyone know how to turn/convert a regular RSS feed into a Media RSS feed? I once found a site which would take your feed and spit it out so that Friendfeed would import the entries with their photos or other media.
From Google Reader shared items or a shared tag
- Mitch
I'm not sure, but I think that Greader shared items produce ATOM feeds. I've never seen Greader shares with multimedia here on FF and I don't think the required elements are included in the feed, so it doesn't make sense to me that there could be a service to add them in to the feed. Hopefully, someone more knowledgeable can chime in here.
- coldbrew
Colbrew - The reader items would have to be added as a blog.
- Mitch
I just looked at someone's Greader shared-feed, and it is an ATOM feed, and it does not appear to include any media enclosures. Check out: http://en.wikipedia.org/wiki...
- coldbrew
I was just wondering the other day what the characteristics of an ATOM feed were. Sounds like media is precluded when using that format.
- Laura Norvig
ATOM does support non-textual content. I would guess it is not included in this instance to encourage people to go to the originating site.
- coldbrew
Media really isn't default in RSS either though is it? You need to start identifying it with MediaRSS tags which is what I need to have "applied" to my GReader feed.
- Mitch
so coldbrew, sorry for my ignorance but what is the difference between a regular RSS feed and ATOM?
- Laura Norvig
ATOM is the IETF backed format (standard), while RSS (now in v2) was the original, and is kind of a de facto standard. On the surface there are no real differences. If you like to read technical content, check out: http://www.intertwingly.net/wiki...
- coldbrew
BTW, Atom isn't an acronym; It's just "Atom".
- Ken Sheppardson
I made a Yahoo Pipe that contained my Flickr feed plus others. When I run the pipe I then get the choice of Image or List - when I get the code for a badge it seems to include all the media contained in the other feeds.
- Nick B.
Nick - can you post a link to your pipe?
- Nathan Rein
First question, why MySQL instead of a straight key/value/attribute store, like a bdb? (Amazon's SimpleDB works along similar principles, btw.)
- DeWitt Clinton
Cool stuff Bret. Love to see this kind of stuff. Would love that chat about the Ajax side of things ;)
- Dion Almaer
DeWitt: Historical reasons; we have some operational experience maintaining MySQL servers, and MySQL is very popular so there's a lot of support available.
- Tudor Bosman
Oh right, none of you are ex-Amazon. (Any Amazonian reading this will understand...)
- DeWitt Clinton
How small do you keep the shards? Can you keep the entire index tables in RAM? Are you backed with SSD?
- DeWitt Clinton
Not all of it is RAM. We use normal hard drives for this system, and we have enough RAM such that the working set fits in memory. Most of the oldest stuff on FriendFeed is not accessed frequently, and that data is not typically resident in memory.
- Bret Taylor
So for something like 'user_id_index.get_all' you'd trust that InnoDB has that user in RAM, and that you have only a small set of active users at any given time (for web requests). Same for the relevant entities you'd "join" against. Makes sense. (Sorry, having a hard time typing coherently tonight.)
- DeWitt Clinton
Yah, those indexes are typically in RAM. Our indexes contain a timestamp as well (since they are ordered reverse chronologically), so the oldest entries referenced by that index would likely not be in RAM depending on how many pages back they are on most feeds.
- Bret Taylor
BTW, did you guys look at any other serialization mechanism other than pickle? Any pros/cons for cpickle?
- Arvind Sundararajan
cPickle is the same format as pickle, just implemented in C instead of Python. We use cPickle. marshal is faster, but according to docs may change from version to version of Python, so we didn't want to use it in our DB.
- Bret Taylor
@Bret "250 million entries", is it possible to give the size of the DB? and what is your opinion about when will you need to change this system again or is it strong enough for years Meybe you will only need hardware instead of structure changes?
- Ömer Faruk Kurt
Ömer: I don't have an exact number off the top of my head, but a lot of data is the indexes themselves, so adding indexes is almost as significant as adding entries.
- Bret Taylor
This sounds exceptionally similar to how the AppEngine datastore does things, except that it's MySQL and not BigTable.
- Alex Power
Aren't commercial RDBMSes (like Oracle, MSSQL) designed to support the sorts of things you're looking for, like online reindexing and joining across shards? Why not use them? Lack of experience? NIH? Too expensive?
- Gabe
"We like MySQL for storage, just not RDBMS usage patterns" Excellent post Bret, thanks for sharing!
- Mahesh CR
Gabe: "designed to support" and actually working are two different things. I think that is why companies like ours generally don't use commercial software and choose to use open source software. If it doesn't work, we can fix it. In my experience, most of those features don't work at the scale consumer web sites need, and it certainly wasn't worth the cost and time to find out from our perspective.
- Bret Taylor
Non-RDBMS-patterns may work when there is no great needs for processing the actual data (like reports etc). In LOB-apps this would be a big no-no, but in large-scale services like FF alternative methods are almost necessary...
- Jemm
Thanks for the writeup! Is there a reason you don't declare the added_id as an UNSIGNED INT?
- Roger
@Bret, this is a pretty cool write u!. Might I suggest as a subject for future write ups, how the realtime update system works (e.g. how it knows what updates it needs to push), and perhaps how the feed fetching system works and how these might inter-relate?
- Ray Cromwell
Ray: good ideas. We will do more of these. We want to in theory - they just take a bit of time to write up, and we like writing code more than blog posts :)
- Bret Taylor
Bret, do you really think that guys like Oracle and MS create features for enterprise-level databases that don't work or scale to meet the needs of those who buy them? Of course you may be right, but the TPC tests generally get results in the hundreds and thousands of transactions per second. In reality, though, what can you really fix if something doesn't work? Are you going to debug MySQL if it starts corrupting data or optimize it if it's too slow?
- Gabe
Gabe: yes, we have already debugged MySQL when it has crashed. It would have been impossible without the debug symbols and source code.
- Bret Taylor
Gabe: I've been doing a fair amount of tracing through MySQL core dumps over the past week, and just having the ability to look at the code, plus the size of the community working with the same code base and debugging problems, seems to make it worth the cost at this point. We haven't changed any of the MySQL code yet, but Google and others certainly have.
- Jim Norris
I'm skeptical of Oracle and other commercial pre-packaged systems because the companies are focused on extracting revenue via software sales and their products are highly optimized for a certain problem domain that doesn't really fit our experience very well.
- Jim Norris
So I guess the downside is that if you change your indexes you need to change the code, too. How do you manage the table creation, code migration and data sharding all at once?
- Nick Lothian
Nick: It is actually not bad: make the tables, update the code, start writing to the indexes for new entities, then run the "Cleaner" to fill in the indexes for the older entities. When the cleaner is done, you can start using the indexes for features.
- Bret Taylor
The most important decision about your design- and why not going with databases RDBMS etc. to manage the indices - is that your users don't really look up the old stuff, so keeping the old entries in RAM/live indices isn't worth the time hit it would take to do huge indexes on long tables. This seems like a "running index"- that is, it's populated by the application in many places at one time, instead of in one place, that the DB then optimizes over the entire history.
- anna sauce
Reminds me of some multi-user java apps back 10 years or so ago that had to manage simultaneous users on live systems. great post, Bret, and fun to see what's working behind the scenes.
- anna sauce
"if you change your indexes you need to change the code" is not a problem unique to this system. You always have to update your code along with your schema. You have to write code to read from the tables even if you use a typical RDBMS, and you can't do that without an index, so this staged process exists in some form no matter what your storage scheme looks like.
- Bret Taylor
Bret, it sounds like MySQL crashes a lot. Are you assuming that commercial products will crash just as often and the vendors won't debug it?
- Gabe
Gabe: Every piece of software crashes, whether it is written by Oracle or by open source developers. The difference with open source is that we are not dependent on someone else to diagnose and fix the problems. This is not a unique sentiment. Google, Yahoo, Facebook, FriendFeed, Twitter, et al, have all chosen open source infrastructure for this and a variety of other compelling reasons we have already discussed.
- Bret Taylor
Brett, I agree that not being dependent on somebody else is great. In fact, the product I work on is probably 99% open source or developed in-house. I even replaced a relational DB with a pickled Python object store a year or two ago. However, I'm also not considering writing my own transaction protocol, indexing, and query engine because the open source DB I use doesn't support...
more...
- Gabe
Re: "change your indexes you need to change the code" - yes, conventionally you need to change your code if you change your tables. But you do get some opportunities for runtime optimisations by changing your indexes (although in practice this can have availability costs as you note)
- Nick Lothian
OK I'm going to put this out there: sounds like you're re-inventing the wheel.
- anna sauce
Being able to open something up to understand, diagnose, and fix things yourself is underrated.
- Amit Patel
DeWitt, part of the reason we didn't use a simpler store is that we're using mysql replication. We'd have to replace that too. Probably not hard, but it's working well for us. Also sometimes you do want transactions for performance within a single DB (each xsaction is a single log write, vs multiple little writes).
- Private Sanjeev
re: SSDs and mysql. I can't speak for all SSDs, but the Intel X25-M gets internally fragmented very quickly, reducing write performance by 10-20x. Things that do large sequential writes like bigtable or lucene are a better fit.
- Private Sanjeev
Part of the purpose behind traditional schema driven designs is to 'protect' the data from the programmers and the applications - which is not insane given the level of commitment people have in some kinds of organization. One of the statements that the FF people are making here is that they have trust in themselves and each other and they care about what they are doing.
- Robin Barooah
You guys rock so hard it HURTS! THANK YOU, FRIENDFEED!
- Josh Haley
Just logged in... this sounds really cool.... hmmm :)
- Susan Beebe
great post -- liked the discussion in the comments too! some people thinking relational databases can do everything; other people not understanding rationale behind key/value store; other people mad you've reinvented it using MySQL... :-D favorite comment from guy who is having a heart attack because anyone might think it's a good idea: "it is like using a database to solve the problem...
more...
- Karim
what is the correct name for this? "schema-less data?" "entity attribute/value (EAV) store?" "key/value store?" personally, i like "property bag" :-D also, have you considered Amazon SimpleDB, or Microsoft SQL Data Services for hosted services, since they are based on similar ideas, just not on MySQL?
- Karim
Sanjeev, I heard rumors about the X25-M performance running down after fragmentation, but last I heard was Intel was unable to replicate & "looking into it." 10-20x should not be hard to spot. :-) Are you in touch with Intel...? Also, have you seen similar problems on SSDs with Samsung controllers? (I went with the Corsair...)
- Karim
Karim, it may not get as bad with other SSDs, but they start off with much higher write latency for small random writes than the X25-M.
- Private Sanjeev
The Samsung, you mean? All I was trying to do was *not* get the JMicron controller and *afford* it :-D
- Karim
but why are the ssIDs varchars, is the question (from @eonarts comment)
- anna sauce
anna: what are ssIDs? If you mean the UUIDs, they are just 16 byte binary strings.
- Bret Taylor
Just look how many comments on this begin with someone's name. "anna:" "Karim," "Sanjeev,". Somehow there needs to be a reply-to-comment mechanism. Fortunately FriendFeed uses json blobs in their database so this won't require a painful schema change ;-)
- Kevin
Bret: you're right, I re-read the post and couldn't find evidence of the Primary Key Erin mentions in her comment, as varchar. Am I missing it?
- anna sauce
schema changes aren't painful, and I have a question for more current DB/programmatic people (than me): why is this called schema-less? There is a schema, it's just that the database application isn't being used to index, Bret's crew are doing it manually. Blobs in columns doesn't mean it's schema-free, that's been going on for 15 years. I think it's just that he's cracking the indexing methodology and offroading. I'd like to hear from some PostGres people on this- Disqus is on PostGres aren't they?
- anna sauce
Anna: that person is talking about the URL index. What we are doing is correct for this application; we want the index to be in that order on disk, not in a different order.
- Bret Taylor
I just remembered that one of my client's DBs is actually Btrieve, which is nothing more than a way of storing binary blobs with certain byte ranges indexed. (They are still using a DOS app from the '80s, with the Btrieve running on a Novell server.) The only real difference is that they have fixed-length keys and records, while the MySQL data is variable-length.
- Gabe
Bret: Thanks a lot for the informative post! Any thoughts on MySQL vs Postgres? We found tables crashed with too many simultaneous writers in MySQL, but your post says it "doesn't corrupt data", so I guess you haven't had such issues.
- Ruchira S. Datta
Bret: Do you apply the same approach to perform searches on your stored text data? (i.e. inside the data storing the BLOB)
- Roger