CJFPM - Google Reader Posts Time normalizer by Share - This component normalize date/time of "share" for order items in date/time on rss. This is an example, clone it and use components in your pipes (unfortunately pipe of pipes not yet aviable :[ ) - http://pipes.yahoo.com/cantorj...
This is awesome stuff, Lucio! I didn't even see that crawl timestamp you converted to the pubDate. I think I'll use this in Feed-buster when a Google Reader feed is submitted :).
- Ivan Zuzak
Btw. what do you mean by "pipe of pipes" ?
- Ivan Zuzak
That every pipe you want to "use into a pipe" (by dropping it from left-menu) have as input a feed or a webpage, you can't use it "in the middle" of an item feed analysis redirecting stream of item into it.. and this limits construction of "pipes as module" http://friendfeed.com/yahoopi...
- Lucio Riccardi
I've just imported this pipe of my Google Reader shares as a blog feed. Is it meant to also pull images from the shares like Feedbuster does? I tried pulling the pipe as a custom/rss feed but it didn't show any entries at all.
- Zulema ◕ ◡ ◕
Has anybody encountered same problem I'm having with the "sort Operator"? Error message I get reads this "Could not determine type for 'pubDate', using 'text' sort." How can I sort (descending/ascending) dates otherwise?
@LouCypher the thing is it used to work and it doesn't anymore. I'm no expert in Yahoo!pipes only copy/paste what others, more proficient than me in that field, do and adjust to the best I can for what I want. Another thing I don't understand: http://friendfeed.com/lelapin... is not parsed when http://friendfeed.com/scoblei... is. How come?
- lelapin
@lelapin can you provide the link to your pipe?
- LouCypher
pubDate sometimes is different for different feeds, and sometime.. there is not a pubDate :) .. i saw this error where there's not a pubDate :)
- Lucio Riccardi
"pubDate" is used in RSS format, "published" in Atom format. You should use YPipes special item "y:published" to sort.
- LouCypher
@zoolcar9 thanks a lot. link_saving: pubDate": RSS format, "published": Atom format.
- lelapin
I think there is a little confusion on RSS pubDate format.. did u notice? I ever seen "2009-6-16T20:40:42Z" from GReader Shared, now i see Flickr use "Sun, 10 May 2009 22:33:38 -0700", RSS specification "Tue, 10 Jun 2003 04:00:00 GMT", again slightly different..this create a little confusion on ordering "my sweet pipes".. then how to manage this?
"In last few weeks i studied some RSS, GoogleReader and FriendFeed protocols and built yahoo Pipes “for fun and profit”. Here the list and the description, take it with some assumptions: "Every pipe you want to “use into a pipe” (by dropping it from left-menu) has as input a feed or a webpage, you can’t use it “in the middle” of an item feed analysis redirecting stream of item into it. This behaviour limits construction of “pipes as module”, then CLONE IT or TAKE IT AS INSPIRATION for your pipe." List of my “public” pipes (consider this as freezed, if I’ll update it I’ll clone a 0.2 version and so on ..): Go to website :)"
- Lucio Riccardi
from Bookmarklet
I want to use a pipe in a pipe, i know is possible, but the input must be a list of items.. Explain: If i have 2 pipes that are able to modify a rss items flux in 2 different way, there is a way to build full modular pipes to use it as components in other pipes? If is not clear i can use examples..
Google Reader / Feed-Buster / Pipes and the order of posts: We all (ahem, me and other 3-4 people for sure) like to use Pipes and Feed-Buster for enhance feed import behavior .. BUT, when you use it on GoogleReader feed, you have to import it as a Rss or Blog, then elements are in "when was published order" and not "shared" order.. (continue)
..then posts are published in "the middle" of the "time-line" of your FriendFeed page, it's not shared to subscribers (is "forget" in the past items, no one see it in real time), Friendfeed don't twitter it.. Then timeline is strictly on item.pubDate? How we can handle it? Smart pipe solution? Challenge is open :)
- Lucio Riccardi
i think i get what you're saying, lucio - FriendFeed sorts the posts by each item's pubDate, which is different from the moment of sharing posts? yeah, i've had similar problems with feeds that didn't have a pubDate at all, so FriendFeed did some weird ordering... anyway, you're right. the pubDate for the greader shared items feed is copied from the pubDate of the feed from which the...
more...
- Ivan Zuzak
but i must say that i haven't tried this out to confirm that this is an issue. are you sure that new items don't come out at the top in FF, regardless of the pubDate? is it possible that items are wrongly sorted only at the moment when you do the import, but all newly shared items are correctly shown at the moment they are shared?
- Ivan Zuzak
@Ivan Zuzak: I don't know .. i think i can make it also with few tricks on pipes (at this time i don't know how but it seems quite possible), but i don't think that reorder with an arbitrary item.pubDate is a big idea.. on every "feed request" can generate also other type of problems.. I imagine posts already-posted that with a date update jumps in the "time-line" like salmon swim upstream :).. need some tests..
- Lucio Riccardi
@Ivan Zuzak: I'm quite sure, results of the pipe is ordered in "Sharing time", on Friendfeed are reordered, I imagine in pubDate time, if i find the exact answer in API I'll write here :)
- Lucio Riccardi
@Lucio: i wasn't thinking of reordering them every time differently. i could do an initial reorder at the moment of import and then internally mark the items in the feed that have been reordered. after that, i just reorder new items which haven't been marked. this is just a conceptual idea :). i too have a feeling that this could be doable in pipes. you can detect the existence of items, insert new items, change old items...etc.
- Ivan Zuzak
@Lucio: yes yes, you're right. i'm sure that the feeds look what we've described and that friendfeed orders the post in the described way. but i wonder what happens in FF when an item appears that has a pubDate which is older than of an item which already exists? how is this new item shown to the user? is it shown at the top of the timeline in the "Home" stream? it doesn't really matter since it doesn't fix anything, i was just wondering how FF handles that from an UI-notification perspective. thanks :)
- Ivan Zuzak
@Ivan: But for reordering in this way i need to manage "memory" in queue, then on every new requent and feed source control compare last time fetched and tagged.. i think we can make it only comparing to a feed reshared from friendfeed.. resemble me a cannibal loophole but it can work (still i'm not sure can be done by pipe.. i'll try)
- Lucio Riccardi
@Ivan: When a pubDate of a new item in a feed is antecedent to last item: the new item is not positioned on the head of stream (and this is not really a problem), is not twittered (haihai...), and don't spread on live coverage of items (haiahiahiahiahi...), because is not on the head of the queue.. :)
- Lucio Riccardi
Thanks Lucio :) Yeah, for my approach you need some kind of "memory". Or do a reordering every time so that the posts at the top of the feed have the newes timestamp. This timestamp may change with each reordering, but it won't matter - nothing will get duplicated or wrongly ordered ever since only the relative position determined by pubDate is important. The actual pubDate is not important, I think. But I'd rather see a Pipes implementation than doing it in my service - since it's less work for me :)
- Ivan Zuzak
On way to resolve this, stay tuned.. :)
- Lucio Riccardi
"When using source modules other than Fetch Feed, the data structure is often not reflective of a RSS structure. When viewing the Pipe as RSS or on the Pipe info page, the results seem to be empty and many users ask on the message boards "why can't I see my data?" So we created 2 new modules, the Create RSS module and RSS Item Builder Module."
- LouCypher
from Bookmarklet
I've been importing a couple of Flickr tags into the groups http://friendfeed.com/umich and http://friendfeed.com/fenderf.... Unfortunately you can't import a tag or Flickr pool directly in the same way you can import a user's feed, so I've had to fall back on RSS. This means each picture shows up as an entry, and the group tends to get flooded. Is this something Pipes could help with? Aggregate the photos and create a single "digest" entry?
- Ken Sheppardson
i don't think you can do this with pipes alone. for aggregation you will need some mechanism of storing state, which does not exist in pipes. for example, one way to do this would be to fetch the original feed every 30 minutes, aggregate the new posts into a single post, and publish it to an output feed. but you'd need to know which posts have been aggregated and which haven't, at least. this could be doable with a cron job in an appengine service, i think.
- Ivan Zuzak
THIS IS A WORK IN PROGRESS As title say, workflow: v0.1 - Images enhancer passing trough Feed-Buster (http://code.google.com/p...) - Remove "[feedly]" from title - Remove html tags from item.comment to post it as a plain comment. TODO: - enrich of item.comment if not present (i don't know how to implement "if then else" or a "case") Comments, suggestion and help are welcome :)
- Lucio Riccardi
hey lucio. i like what you're doing! in the next version of feed-buster i was hoping to implement that "enrich with comment if it doesn't exist" feature, and a bunch of others. can you give me some pointers on how you'd like that feature to work? i was thinking of scraping the first X characters from the contents, where X could be user-defined and would default to let's say 450 chars. if you have any suggestions on how to make the service better, or have any other issues with importing feeds - drop me a DM.
- Ivan Zuzak
btw. if you're poking through the repository of feed-buster - it is currently a big mess, sorry about that. :) i'll tidy it up a bit in a few days. let me know if i can help with feed-buster or with pipes.
- Ivan Zuzak
(add me so i can send you DM :D) .. by the way i don't experienced strange behavior of service in last hours.. I think item.comment need 1) html to plaintext, 2) limit (better if customizable but 400-450 is enough) 3) in some cases item.comment dont' exist, find the right choice. On the last point. I don't know how can be "strict" RSS implementetion but i see on Pipes parsing a mess...
more...
- Lucio Riccardi
All sounds good. :-) Better media support (video not just images) is always welcomed. :-)
- Kol Tregaskes
@LouCyper: humm i'll watch it again.. but i need if-then-else to control a variable, and if is empty then copy content from another variable, and "F i l t e r" as i see have only a pass/block for post on certain variable rules, correct me if I'm wrong.. we need: if [item.comment==""] then item.comment=item.description; .. seems easy but i don't saw nothing like this on pipes help/examples.
- Lucio Riccardi
@Edwin Khodabakchian: I know but i wanted to consider the base case .. it's a generalistic pipe
- Lucio Riccardi
Lucio - hers's how you could possibly convert html to text. Use a loop operator and within the loop operator a String Regex module. in the string regex, specify a regex that finds all html tags and removes them. a simple (but incorrect) solution would be to replace "<.*?>" with "" (empty string). for an explanation why this is incorrect have a look at http://nadeausoftware.com/article.... but this simple approach should also for for most cases.
- Ivan Zuzak
no idea on how to implement the if-then-else you want in Pipes. i've developed a dozen Pipes and found that they are awesome if you want to do exactly the things for which the modules were intended for. however, if you want to do something different, or more complex, you're in for an unpleasant ride. have a look at YQL and YQL execute statement, the web service module in Pipes, and AppEngine for a bit more freedom.
- Ivan Zuzak
For html to plaintext i resolved in the exact same way you suggested :) .. for if-then-else I'm far from a solution... however in few hours i have a big news for FriendFeed/GoogleReader/Pipes lovers :)
- Lucio Riccardi
wicked Lucio! can't wait for your news :)
- Ivan Zuzak
I'd like a "which tab is playing that unbearable music" feature
- Chuck Baggett
You do realize plans to implement this have already been announced, right?
- LANjackal
That's funny, I remember when they were against the idea.
- Chris Charabaruk  ̄ω ̄
And given the timeline for implementing this in Mozilla, it's unlikely to happen before Firefox 5. Best to move to IE8 or Chrome if you really need this.
- Chris Charabaruk  ̄ω ̄
Chuck: tou are totally right!!! :) (tnx god Facebook, Friendfeed and co. avoid it completely)
- Lucio Riccardi
Chris: Yes it's scheduled .. chrome is not on my preferred OS (Linux MacOsX) (humm... god know why?), nor IE8 (when i work on vista i don't see a lot of reasons to not use Firefox for IE8, nor javascript/processes shine behaviour)
- Lucio Riccardi
I want to build a yahoo pipe that extract one image link from "item.content" and assign to "enclosure.url" to see it in friendfeed import... but i'm stuck at the content parsing .. regex? suggestions?
"This post is part of the Who's @ Google I/O, a series of blog posts that give a closer look at developers who'll be speaking or demoing at Google I/O. Today's post is a guest post written by Paul Donnelly and Jonathan Trevor of the Yahoo! Pipes team."
- LouCypher
from Bookmarklet
"Output similar to that of standard Flickr user photostream RSS, but allows up to 500 photos to be retrieved instead of the standard 20. More info @ http://flickr.com/groups..."
- LouCypher
from Bookmarklet
I created a Yahoo Pipe that will give you a feed of only new topics you created at GetSatisfaction.com. Things you asked, shared, reported, and gave: http://pipes.yahoo.com/pipes...
"Two years after it launched, Yahoo Pipes is still providing an amazing way to mash up data from around the Internet. Ars looks at Pipes and introduces a step-by-step walkthrough of some of its functionality."
- LouCypher
from Bookmarklet
Has anybody tried to do some calculations with elements extracted from a page? Of course said elements should have the proper numeric format in order to proceed on with math manipulations. I'm interested in making up and following my own portfolio and am currently facing that particular problem.
"The Yahoo Pipes “expert” Hapdaniel (y!) (twitter) tells me I’ve fixed nothing, and should have read his tweet. Also, I’m apparently a terrible person for having a huge master OPML that I want to efficiently process. Too bad he’s too busy being an “expert” to notice that I added the ability to “filter on folder name” while scanning the OPML! My feedback to the nice people at Yahoo: I tried to find suitable pipes, but in the end I had to write them myself. In addition, the person who responded on the message board was most unhelpful and nasty, and obviously thinks it’s a terrible thing to have a large OPML. And, he doesn’t understand that one wants to filter while scanning the OPML - that’s a key to having a huge master OPML that you can still use for multiple things efficiently. I doubt I’ll participate here much going forward. I like Pipes though, and blogged about it, and deployed them on my site… Thanks… and nice job on the Pipes machinery itself."
- Richard Walker *exculted*
from Bookmarklet
"Here is the published Pipe that powers my Recent News widget on Soft Machine: OPML feed items newest 20 with [source] prepended There are a few details I don’t need to go into, at the moment… One thing I will say, however, is that the aggregator pipes I looked at either didn’t work or were extremely inefficient. Others weren’t suitable, or were fragile, or very specific to a task. This Pipe is general, efficient, and quite good at producing recent items from OPML files. The only ongoing upkeep required is to re-export my subscriptions to the cloud when I make changes I want to see reflected here! In all, I’d say the capabilities of Yahoo Pipes are good, but you may not find what you need easily."
- Richard Walker *exculted*
"Here is my published OPML items pipe. Here is the Y! message board thread. This is the first step in solving what I consider to be the distressing state of OPML support in Yahoo Pipes. A very simple fix, it consists of setting up a pipe that parses an OPML file (such as you find when “exporting” subscriptions from Google reader), and outputs the subscription items. I take care to handle both “flat” and “structured” OPML files. Note, I recognize the “flat” items by filtering on “item.type” having “rss” or “atom.”"
- Richard Walker *exculted*