"Nicely done! I can't wait to see how smoothly this ends up including other pieces of a distributed 140 character system... and more! Then the fun really happens. :)"
- Jeremy Felt
"Hi Brett, thanks for stopping by. This comment stream aside, the original post was written more as my perception than science. Rereading, it's a pretty unorganized perception. Ahh, late nights. There's more too the rambling, but if you come away with one thing from the above, it's that I don't see the FeedBurner stuff being real time as I thought it would. I'll be grinding through the data more closely as the week goes on. The initial conclusions are based on a snapshot look at the initial 24 hours or so of use. I can't answer to the latency yet, but I also can't imagine it being too high. Not a perfect answer, I know, but the server is on Amazon's EC2 and overall latency (network and system) seems low. Almost the only traffic coming in is from rssCloud and PubSubHubBub notifications. From watching Dave's rssCloud log (light pings), the time posted is usually less than .300 seconds. The feeds that I've noticed the most issues with are from FeedBurner. My guess is that the delay and..."
- Jeremy Felt
"Hi Brett, thanks for stopping by. This comment stream aside, the original post was written more as my perception than science. Rereading, it's a pretty unorganized perception. Ahh, late nights. There's more too the rambling, but if you come away with one thing from the above, it's that I don't see the FeedBurner stuff being real time as I thought it would. I'll be grinding through the...
more...
- Jeremy Felt
"Google did make announcements though. http://adsenseforfeeds.blogspot.com/2009...... - Google announces PubSubHubBub support in FeedBurner feeds for AdSense, notifying a "Google-run Hub". http://googlereader.blogspot.com/2009...... - Google announces PubSubHubBub support for shared items in Reader. http://buzz.blogger.com/2009...... - "All blog post feeds now contain a "hub" element, and will ping Google's hub on every post update." http://googlecode.blogspot.com/2009...... - "we have gone a step further and added PubSubHubbub support to Google Alerts." I'm not trying to push any blame on Brett. Google owns this now and should help any issues along. I've written about some of the issues I'm seeing with Google's PubSubHubBub Hub. Constructive discussion about the issues I've been seeing is definitely welcome."
- Jeremy Felt
"Cool, I'm not a big AppEngine guy. Not trying to argue the architecture. My perception is that it's stable and stays up. Plugging the hub into many feeds is a great way to test. But, two things. 1) When you're big like Google and you announce the implementation, the perception given to me, the developer, is that you're ready. 2) When I, the developer, do start using it, I'll get a perception on how it's working and I'll share it. If others have details on how it's working for them, I'd love to share examples. I'm having fun working with both rssCloud and PubSubHubBub and coding > arguing."
- Jeremy Felt
"Yes, referring to the official PubSubHubBub server that Google employee Bret Slatkin wrote and deployed. The one that Google then Pushed to their Blogger feeds. Got it, no multiple hubs. But by using multiple IPs from the AppEngine network, a distributed network and uptime is inferred. My point is - if you're going to push yourself into millions of feeds as a solution, then you are ready to go. BTW, I'm not arguing against PubSubHubbub here. I want it to work. That's all that I'm getting at."
- Jeremy Felt
"The two circumstances are different. There are always rough edges, but.... The rpc.rsscloud.org server you are probably referring to is maintained by Dave with no promise of uptime (correct me if I'm wrong, obviously) as a place to test your implementation. It is possible that the server can be rebooted for changes at any time. No big company is providing a constant connection here. The WP plugin for rssCloud creates a server on every blog that installs it. Problems have been few and far between with this. The pubsubhubbub server hosted by Google has been pushed into every blogger feed and implemented heavily in FeedBurner feeds by Google. It pushes updates from multiple IPs, indicating a network of hubs that they are using to guarantee uptime. By doing this, they have told me that they are ready for real time."
- Jeremy Felt
True, ~18GB per tweet if all 2.2mil ran their own aggregator. Much lighter than the 100s of GB/day sites such as Gawker from pageviews alone (I think). And she is Oprah. :)
- Jeremy Felt
The scale looks so much better when you've got a fat ping in the picture. With a well-encoded fat ping, you could probably get each of them under 1kB. That's not much more than the size of the HTTP request required just to inform the reader that something has updated - and that's all that we need to send them. An order of magnitude is a pretty significant savings.
- Matt Mastracci
I definitely see the benefit to a fat ping. I'm just having a hard time visualizing the scalability problems that others seem to see right away with rssCloud. There should be virtually the same number of "grab entire feed" requests as the current polling based stuff, just without the "have you changed" requests that normally go along with it. With either method traffic should flow a lot smoother-- but I am digging the option who's center point seems to have little less stress on it.
- Jeremy Felt
Yeah, I think PSHB got a bad rap as "complex" early on and it's having a tough time shaking it. When you implement it at the place your feeds are generated, it's trivial (you are generating the same RSS channel/Atom feed, but with the single item). If you want to outsource it, you pull something off the shelf and talk to it. I've been so busy with building our product that I haven't had...
more...
- Matt Mastracci
The thundering herd issue is really just one piece of the puzzle, anyways. The comparisons between the two protocols have been done to death and it's time to code. :) I hope to have some cool real-time feeds come out of some of the work I'm doing now.
- Matt Mastracci
Lol, you're right about that. Most of the time i just shake my head and keep typing away. Today I just had to ask. :)
- Jeremy Felt
"Looks like it's still propagating. When I do a "nslookup r2.ly" from my Windows machine, I get my openDNS resolver, which returns 208.69.36.132 when I ping it. But when I use this (http://network-tools.com/nslook...) to do an nslookup, it sees that slicehost has it. Should probably fix itself soon. Assuming that you've gotten word within the last few hours, of course. If it's been a day, then I'd start to wonder."
- Jeremy Felt
"I'm probably confusing things by not being completely finished with it yet. :) That subdomain didn't exist until last night. I added "rss" as a TXT record first, and then later as an A record. I didn't get around to doing the redirect - was having too much fun with other things. Right now it goes to a "tesT" page on the server, but will redirect later to my feed."
- Jeremy Felt
"The main purpose is to eliminate confusion, which came up quickly in the previous DNS post. We all plug the address in our browsers to see what happens because seeing the TXT record isn't as easy."
- Jeremy Felt
"Awesome! Just tried it and it works on my previously registered supercloud already. Good, good stuff. I'm going to go add an A record for rss.jeremyfelt.com that I setup a TXT record for after your earlier post. I probably overstate this, but I have so much fun with tech that slides together well."
- Jeremy Felt
"I like that idea, hadn't thought of a combo deal. :) Anything where the user only needs to know one thing (their address) and the technology (browser/cloud app) figures the rest out seamless is a good way to start."
- Jeremy Felt
"It doesn't make a website or redirect to your feed, it only adds a DNS TXT record for the feed specified. Once you have this, an application that is looking for TXT records will know how to translate connectme.supercloud.org to whatever feed is stored as TXT."
- Jeremy Felt
"The practical applications are pretty clear to me. You can store your feed in a TXT record under a domain name. If you move from 140char Client1 to 140char Client2, you can change your TXT record from feed1 to feed2 and the aggregators that are paying attention won't be required to notice the change. They'll just deal with it. You can change services without your friends having to follow around manually. As for handling situations where other TXT records exist under that domain... It's up to the aggregator to know what a feed is/isn't and if it is/isn't cloud enabled. We'll just grab them and check."
- Jeremy Felt
Cool. Thanks so much for this. I just imported this OPML into my reader. (Feedly)
- kevin fitts
It worked? I'm glad they can handle the bizarreness in that OPML. I realized I could add documentation to it and that most aggregators would just ignore anything that wasn't a feed. Tell them to support reading lists, so when I add feeds to that OPML they'll pick it up for you. I just added a television feed this evening in prep for the Emmys tomorrow.
- Dave Winer
This imported beautifully into My Status Cloud. I'm doing basic stuff though, not rechecking the feed or storing the URL yet. Soon to come, soon to come. :)
- Jeremy Felt
I love small groups of developers who are working together to make the web better like this meeting of a bunch of developers. At top is Matt Mullenweg and Dave Winer. This is happening now at Berkeley. Will report more later.
- Robert Scoble
from email
I believe Twitter uses HTTP polling from the client and rate limits it to 150 requests per minute
- Thomas Beutel
Can someone there please tell them they need to fix the IP issue (edit - I see Matt has already mentioned it)
- Nick Lothian
My 2 questions still are 1) How do we make it easy to run inside a firewall and 2) Can servers really scale, especially when I would suspect the potential for "spanning clients"
- Bill Grant
Nick, yeah, it came up early in the meeting (on the audio)
- Matt Mastracci
that should have been "spamming clients"
- Bill Grant
Although noone brought up the specific "datacenter" issue... @glenc also mentioned it was an issue for Yahoo!
- Matt Mastracci
There's also a lack of subscriber verification that needs to be brought up.. all you need to do is get someone to say 200 OK to this protocol and they'll get spammed for 25 hours.
- Matt Mastracci
I must be old fashioned but as a client I prefer asking information and getting it, rather than for the potential of servers all over looking for me
- Bill Grant
WP rssCloud had an issue in 0.2 where you could specify a path of "@otherserver.com/path" and it would subscribe that other server. (fixed in 0.3)
- Matt Mastracci
Someone ask about integrating activitystrea.ms into rssCloud
- Jesse Stay
Jesse - your short domain got redacted too ;)
- Matt Mastracci
Ustream needs to fix that, or we're just not going to post over there
- Jesse Stay
PSHB doesn't have the last mile problem to the same extent as rssCloud, because you can choose IP _AND_ Port to subscribe, which will work better for NAT traversal.
- Nick Lothian
Robert, that's Joseph Scott from Salt Lake City - tell him hi for me
- Jesse Stay
what is Facebook using for their updates? or are clients like Seesmic just polling?
- Thomas Beutel
Jesse - I've had it work before. But even a basic search online isn't allowing links through. Normally moderators/admins have permissions under that widget by the send button in Ustream. They changed the defaults about a month ago - and nothing shows under the settings.
- Courtney Engle
Courtney, yeah, it's a pain in the neck - I've had the same problems with shows I've administered
- Jesse Stay
I'd like to hear more about naming (.tel?). I was also thinking about reverse lookups, i.e. resolve the feed from the identifier and vice-versa (for the benefit of the cloud and feeds it produces for indexers etc)?
- Andy Chantrill
discovery for atom is very easy ... - I see Matt M. just posted the info
- bear (aka Mike Taylor)
Why are they discussing a new namespace? It's already supported.
- Matt Mastracci
Maybe not now, but I'd really love to see some discussion on activitystrea.ms - if you want Facebook or MySpace that's the way to get them in
- Jesse Stay
I noticed Feedburner hid the places they ping a few months before PSHB came through. Wondered why that was put threw a few pings into my WP sites manually
- Courtney Engle
I think the discussion here is almost just as interesting (and possibly more) than what's going on in the room. I wish we had more of them participating.
- Jesse Stay
Yeah, it kinds of sucks that there's a wall between the two. Something to keep in mind for future meetings
- Matt Mastracci
BTW, Robert, Joseph Scott (the guy from SLC) is the guy that wrote the rssCloud Wordpress plugin. He works for Automattic.
- Jesse Stay
@bear - why do you say that? A rssCloud consumer just gets the whole feed. Obviously it needs to know what to do with it, but that's the case anyway.
- Nick Lothian
And aside, seriously PSHB > rssCloud for many reasons. I don't see a reason to create two "standards". I'd LOVE to understand why Wordpress opted for rssCloud.
- directeur
isn't activitystrea.ms format a different namespace than rss and/or atom?
- bear (aka Mike Taylor)
Good luck with Bluehost - I know people that work there and their service is horrible!
- Jesse Stay
Matt - a wide angle webcam would be spiffy for future meetings - or TalkShoe/BlogTalkRadio to just use audio only works well.
- Courtney Engle
nick - wait - so an rsscloud update is the whole feed?!? not just a single payload item of the latest history?
- bear (aka Mike Taylor)
Nick, I don't think so, it just gets a notification, not the feed
- directeur
bear: rssCloud subscribers get a simple ping (POST url=blah) and are expected to scrape the whole feed themselves.
- Matt Mastracci
Matt - thanks for setting up ustream, good to listen in to this discussion...
- Bill Grant
rssCloud is a ping, and the client goes out and retrieves the whole feed I believe.
- Jesse Stay
@Nick when PSHB does that, that is from the publisher to the hub? or all the way to a single client?
- Bill Grant
I just asked Matt why he went with RSSCloud. He said "it seems like a good thing to do." he also said they will do PubSubHubub too.
- Robert Scoble
from iPhone
That makes sense - I would do both too if I were them. Let the aggregators decide.
- Jesse Stay
Bingo, competition is always a good thing.
- Jeremy Felt
It's a shame that the IP address endpoint question wasn't answered. That makes it a non-starter for Google, Yahoo and anyone else with a distributed datacenter.
- Matt Mastracci
I'd love to also see more support towards OMB - I really like that protocol
- Jesse Stay
@Bill Grant - it's only from the hub to the client. It retrieves the feed from the original server in the same way as rssCloud (ie, gets a ping, and then grabs the feed). However, you can chain PSHB Hub servers together, so only one needs to respond to the ping (not sure if rssCloud supports that or not?)
- Nick Lothian
just had a rush of ideas - man I need a day job that just lets me code on social web stuff.
- bear (aka Mike Taylor)
@matt - your blog post is almost the same as I was thinking. xmpp pubsub back-end with a PubSubHB and/or rssCloud front-end
- bear (aka Mike Taylor)
bear, yeah. I implemented a gateway from Pubsubhubbub to XMPP this morning (last night?) as a proof of concept here: http://pubsubhubbub-xmpp.appspot.com/ It's unlikely you'd get that working in rssCloud because of the subscription API limitations, thought.
- Matt Mastracci
@bear SRV records for subscription endpoints? So if I'm plaggypig.tel, I could just set up a bunch of records for all the aggregators/services that I want to use, rather than the cloud assuming it should be the source of my request?
- Andy Chantrill
andy - I was mentioning SRV as the way to broadcast the XRDS or other list of endpoints on your service. This gives a way for people behind firewalls to advertise internal services. But I could also be completely not grok'ing how rsscloud info flows.
- bear (aka Mike Taylor)
@bear Why use XRD documents when DNS is right there? $ dig rss.andy.tel NAPTR; I thought you meant using SRV records as well to define my subscription endpoints (I could have several).
- Andy Chantrill
andy - then it sounds like i'm making an assumption that isn't matching reality - I still haven't done a test implementation yet of rsscloud so apologies if i'm clouding the issue
- bear (aka Mike Taylor)
bear - rssCloud and Pubsubhubbub are both based on opt-in (via HTTP POST) notifications that happen via HTTP POST whenever content is updated or changed. The publisher notifies the hub via HTTP whenever content changes, and the hub then notifies the subscribers.
- Matt Mastracci
Just realized that the IP restriction thing means rssCloud can't work with virtual hosting (ie, anywhere more than one domain is hosted on a server). Found that out when trying to implement it. Yay.
- Nick Lothian
Great now I have many more questions than answers. Thanks for the conversation on the details of the distribution systems gents.
- Mark Essel
Re "One thing, though. I actually like the bigger photo and no map." -- I added a setting on the site integration page to make thumbnails be max FriendFeed size (height 175px). You can always turn off maps with phone options.
- Bruce Lewis
Jesse: yup, Wordpress.com has six million blogs as of May 2009 and is growing quickly, so that's a lot of blogs that just got RSS Cloud.
- Robert Scoble
Robert: Are you going to be at the rsscloud meetup this week with camera in hand?
- Jeremy Felt
I guess Alex Conner of (Geekery For Rent- @gk4r ) and I are disruptive because we worked on a site that has to launch this Wednesday. :)
- Melanie Reed
Robert - great blog post - CNN are you listening?
- Susan Beebe
I got a quote and bought homeowner's insurance on Labor Day. Hey, GEICO was open so why not?
- Morton Fox
Jeremy: not sure I'll get there. Will try. Too much to fit into one week!
- Robert Scoble
Robert: Cool. I'm thousands of miles away, trying to find a way to get audio/video. :)
- Jeremy Felt
Whoa. One big problem I see with rssCloud: "By convention registrations expire after 25 hours. Workstations should register every 24 hours for each subscription to keep them current." What if I'm a program like Google Reader or FriendFeed? Suddenly, if I want rssCloud support, I literally have to ping every site on the entire internet, daily. Seems wasteful to me.
- Otto
Pinging once every 24 hours and then being told whenever there is an update instead of having to ask every 10-30 minutes if there has been an update should be easier and possibly less expensive.
- Jeremy Felt
"We’re also going to be supporting other ways for people to get push notifications (Jabber, email, Weblogs.com pings, SUP, pubsubhubbub, Twitter… who knows what else)" Great news
- Steve Gillmor
This definitely puts the nail in the coffin of the "RSS is dead" contentious debate. Now we can focus on innovations in-and-around RSS. (In response to the Google Reader question, I think that Google will use their HubPubSubHubb in conjunction with Feedburner,- however I suspect they will need to be compatible with rssCloud) Translation for end-users: wait and see.
- William Mougayar
from FriendFeed MT Plugin
@William Hopefully - its a pointless debate - always was.
- Chris Saad
pointless for those who don't care about realtime
- Steve Gillmor
lol saying that to me is kinda silly too Steve :)
- Chris Saad
Instead of debating, why not just let the API mashup devs do their thing and the rssCloud (RSSCloud?) devs do their thing. We (the social media megaphoners) need to just shut our speculating pieholes, wait, and anticipate being blown away (or not). Turning non issues into issues is SO Web 1.0. #justsayin#rsscloud#api
- Mona Nomura
it's not a non-issue, and these technologoies were invented to give people a voice
- Steve Gillmor
The fact that we are all here in FF (SUP) vs. discussing via instantized blog/comment volley/query rebuttal is telling. This notion of approaching more realtime options is how blogs might get back to being useful for certain groups of folks. Maybe?
- Jay Cuthrell
Real-time is a very important feature Jay - super important. So are a number of other key features of the social web and a number of new emerging features of the next web.
- Chris Saad
(didn't mean to inject or hijack anything there btw... this is all good stuff)
- Jay Cuthrell
Thanks, Dave. Congrats on the big news, btw!
- Mona Nomura
To add - if "real time discussions" are pertinent to your forward thinking, innovations, and products: You're Doing it Wrong. Form opinions, then discuss. If you're wrong, apologize, learn, and move on. #CriticalThinking101
- Mona Nomura
In hindsight, Steve's obsession with the subject of real-time might have heightened our attention on the debate, but also perhaps hastened the development of rssCloud. I'm sure that in the back of Dave Winer's mind, he wanted to show that Twitter (and Friendfeed to some extent) wasn't the only real-time game in town (in addition to the fact that this hook was already available in RSS...
more...
- William Mougayar
from FriendFeed MT Plugin
William, you'd be 100 percent wrong about that. I was doing this stuff before Twitter or FriendFeed existed.
- Dave Winer
I'm not so sure Twitter was ever real time. Only centralized. This made it appear realtime. (Except for the lucky few - i.e. friendfeed)
- Jeremy Felt
reagrdless of the prior art discussion, certainly dave's embrace of realtime is good news
- Steve Gillmor
@stevegillmor: prior art? What on earth are you trying to say in this context? Since when did prior art have anything to do with real time? Or have I missed something fundamental?
- Dennis Howlett
Let's see what happens with all the million Wordpress.com blogs. But I have a felling this will be big but we won't see the impact for a few months
- Wayne Sutton
from iPhone
Dave, I'm glad I was wrong (as I hinted to the fact that you had that hook there from 2001). I recall well how Radio Userland and Manila used to interchange feeds in real-time. You sure have kept us in suspense, til now, though!
- William Mougayar
from FriendFeed MT Plugin
"Interesting, while the main Salon blogs don't seem to be supporting rssCloud yet, there are a few that appear to be running Radio UserLand still and they have it enabled (probably since they fired up). Haven't tested to see if it works yet as I'm still working on my xml-rpc implementation. Going to try it soon though. :) List here: http://blogs.salon.com/"
- Jeremy Felt
"Yeah, I loved it when I added them to my river last night. I actually started randomly going through blogs I found here-- http://botd.wordpress.com/ -- and adding what I could find. Was surprised by some of the bigger names."
- Jeremy Felt
"No problem. I did the same thing for my first many attempts before I understood the message about not being behind a firewall. You're right, the log is very useful."
- Jeremy Felt
"CNN wire is here: http://rss.cnn.com/rss... I had to find the very small icon, then click on "View as XML" in FeedBurner. Not too friendly. :) The political ticker was a bit easier, just a small icon: http://rss.cnn.com/rss... Nudge harder. :) I half expected them to be flipped on already yesterday."
- Jeremy Felt
"For REST, you make your request through the server specified in the cloud element of the feed you are subscribing to: http://cloud.domain:cloudport/cloudpa... You pass parameters telling the server how to notify you: notifyProcedure=na port=yourport path=yourpath protocol=yourprotocol url[n]=urltosubscribe The server then notifies you on your public IP address at the port and path specified using the protocol specified."
- Jeremy Felt