The first five bit.ly items I Tweeted today would rank 2, 2, 3, 3 and 3 on http://jay.40twits.com/ Now either they all took off like a shot, consecutively, one Wednesday morning, or there is something I am not getting, or....or what? - Jay Rosen

And now that http://jay.40twits.com is back up again, I just Tweeted a typical @jayrosen_nyu link and the normal numbers for tr.im are seen, about 3 x lower than the bit.ly rate for number who clicked on it. - Jay Rosen

Okay, some numbers, this latest link http://twitter.com/jayrose... is at no. 30 on the top 40 chart, with 137 clicks on the tr.im system. This is normal for an item like that on http://jay.40twits.com/. A comparable item like http://twitter.com/jayrose... is now at 587 clicks on bit.ly, which would rank no. 2 on the chart. See what I'm saying? - Jay Rosen

I would suggest you put Google analytics on the target page and find out the REAL metrics - Ric Johnson

Bit.ly also counts other shortlink versions of the same URL. Perhaps people are re-tweeting you link, but only after they visit the page, and using another url shortener? Google analyticsmay help sort this all out, but you're still not going to know the origin of the traffic since most twitter traffic comes from third party clients. - Mr. Gunn

Bit.ly separates the clicks off my link from the total clicks off that url from bit.ly users. At least it says it does. - Jay Rosen

I'm with Ric. Real metrics answers your question authoritatively, and that's good info to have. But, I'm not sure I see the whole picture clearly. It doesn't sound like apples and apples. Doing a blind test is difficult if you're relying on a single source. Doing multisourced links is only meaningful if you can provide impression stats. I'm pretty sure you can't do that with Twitter (unless you know someone or something I don't). - Jason Nunnelley

Have you checked the referrer stats via bit.ly to see what it looks like? Maybe some site is picking up your bit.ly tweets. - Jeremy Felt

I don't get how Google analytics helps me. I must be missing what you are saying. I am not trying to find out how many people came to a given page, I am trying to track how many people USED (clicked) the link that I selected an sent out to them. Because of Re-Tweeting and switched url shorteners, this is not exact, but both tr.im and bit.ly claim to measure the number who clicked the unique URL created for me and my Twitter feed, and if that url was Re-tweeted, those clicks are counted too. - Jay Rosen

I'd agree with the other folks on here...it's probably somehow not an apples to apples comparison. The other option (oh, I see Jeremy just mentioned a form of this) is that bit.ly links are being "watched" more closely by some site/service/tool. - Ken Kennedy

In this example http://twitter.com/jayrose... the unique url is http://tr.im/o2in which is currently no. 24 at http://jay.40twits.com/. I guarantee you, if I used bit.ky it would tell me that three times as many clicked on MY link. And what I am asking you is: why? You're answer seems to be: apples to oranges. I say: it's apples to apples for the user. Both services claim to do the same thing. I think... - Jay Rosen

I'd say that their claims, while sounding identical, are probably somehow not, Jay. One could be stripping out multiple clicks per IP address w/in a short timeframe or something, for example. (intended to prevent double click counts, but would also hide multiple users behind a NAT, for example). - Ken Kennedy

Ken is almost certainly right. tr.im isn't counting some traffic that bit.ly is counting. I'd bet on tr.im's number being the better reflection of actual interest in the link, based on other data I'm seeing, like hit counts on blog posts linked to from my twitter feed, or flickr pics. their counters usually pretty closely track tr.im's. - Dave Winer

BTW, this is great stuff. I wanted Jay to lead a technical investigation to give him an idea how we do it in developer land. I expect this will be discussed on a future Rebooting The News podcast. - Dave Winer

Hmmmm. Okay, so when you say apples to oranges the referent for that statement is the counting mechanism several layers underneath what the site says to users? As a user, "apples to oranges" means: "Jay, you're reading the claims of the two services wrong....." But I say I'm not. It's only apples to oranges for the coders, right? - Jay Rosen

Good data, Dave. That'd be a decent way to check...choose two links that you can see the backend data on (ie, flickr photos where you can see how many times it was viewed, or two blogposts at PressThink, where you can see the server logs.) Then tweet one w/ a tr.im link, and the other with a bit.ly link. Then compare what your server logs say, hit-wise, to what tr.im and bit.ly say. The difference in magnitude of these numbers (3x-ish) should make the "correct" data fall out pretty easily. - Ken Kennedy

Yes, Jay...the "apples to oranges" comparison was in reference to the claim that the claims are identical, if you see what I'm saying. I'm saying I bet they aren't. They're counting different things, but the difference is subtle enough (technically) that when they reword their claims into "non-techie", they say statements that sound identical. Did that make more sense, or was it still unclear? - Ken Kennedy

Funny observation. I (with barely any followers) posted one Trim link and one Bitly link seconds apart to the same URL with the same description. Trim climbed to 12 hits immediately, bitly stayed at 4. I think Ken/Dave are right. Jay needs to post a url through both services to something where the real data can be watched. Either lots of Jay's followers dislike Trim and refuse to click links, or something is being piped out or counted multiple times somewhere. - Jeremy Felt

IN-teresting, Jeremy. Interesting. - Ken Kennedy

Aha, so I do have a mystery of my hands, eh? Ken: I get the oranges now. Thanks. - Jay Rosen

I'm just a "dumber" user; I don't know how any of these things work inside. But I can't post a link to the same Joe Klein post at Swampland, through two different urls; the first one uses up many of the people who would click on the second! A second pass through the same people isn't a valid measure. Followers don't want the same thing twice. So I don't get why that would help. - Jay Rosen

Two different posts, not one. What you're comparing isn't tr.im to bit.ly, you're comparing tr.im to a known value (the server logs) and bit.ly to another known value (the server logs for a different post). One is probably very close to the log values, and one isn't. - Ken Kennedy

There do appear to be differences in bit.ly and tr.im if "a user is not logged in" (which I take to mean if a shortened url is generated by an non-logged in user, not if it's clicked on by one). tr.im says they generate unique tr.im urls for EVERY person who creates one; two different non-logged in users generating a tr.im of the same url get different tr.ims. Bit.ly apparently generates the same bit.ly url for all non-logged in users. Let me try something... - Ken Kennedy

I am accessing tr.im through Dave's app. http://jay.40twits.com/ in m toolbar. But I am logged into bit.ly. So I am supposed to be logged in all the time to tr.im? It's generating a unique url for me, Jay Rosen, it says, and Dave turns that into http://jay.40twits.com/ Maybe I have to stay logged in to tr-im all the time... - Jay Rosen

OK...I take your bit.ly example url from the tweet you mention above (tweet: http://twitter.com/jayrose... - bit.ly url: http://bit.ly/vYfRX ). Clicking on that link takes me to the Friendman article http://www.marketwatch.com/story... So far, so good... - Ken Kennedy

When you use my bookmarklet you are logged into tr.im. My app is logged in on your behalf. You do not have to be logged into tr.im. - Dave Winer

I open Chrome (separate browser, separate cookies, etc.) and go to bit.ly; not logged in (I don't even think I have an account). It gives me an url: http://bit.ly/DjpLP That's good, it's different than yours. It DOES give me a way non-zero click number, though...730 at time I created it (2 minutes ago). What number do you see in bit.ly for that Friedman link right now? - Ken Kennedy

OK, fascinating. If I read the bit.ly site right (http://bit.ly/info/DjpLP), there are (currently) 743 "aggregate clicks" to the destination URL (at marketwatch) that have gone through bit.ly in total. Your specific version of that link accounts for 643 of them (http://bit.ly/info/vYfRX). So there does appear to be a difference between the "not logged in" and "logged in" accounting. - Ken Kennedy

back to apples to apples then because I WAS logged in at bit.ly and Dave says I AM logged in on tr.im What I said earlier was: since I am using Dave's app in the toolbar I didn't know for sure whether I was logged in at tr.im, but I know I was logged at the bit.ly site. - Jay Rosen

But it only shows 10 clicks in the past hour-ish (there's a couple of time errors, but I think they're just typos; it says 12:51am, and means 12:51pm, and it's EDT, not EST. I don't think either of them are any problem.) You posted that link around 4 hours ago, right, Jay? That indicates something HUGE happened very early on. I wish we could see detail click info further back than the "Now" tab shows in the info screen...I wonder if there's an API way to go back further. - Ken Kennedy

re: apples to apples. Not necessarily. *grin* There are lots of apples to apples going on here. We just got rid of one; the logged in issue. That's good! But if tr.im is not counting individually counting multiple clicks from same IP address w/in a short time, for example, then you've got an issue. Welcome to bug squashing, tech style. *grin* - Ken Kennedy

Funny you say that: because I notice that the "immediate" rush from bit.ly was much huger. But it takes 5 mins or so for the first readings to show at http://jay.40twits.com/ so I don't know....I have not been using bit.ly since 40twits came back online, so it would not show much now. Besides: which count is right? what I really want to know is: is this a quality bug or some garden variety thing? - Jay Rosen

Ok, I poked the bit.ly API (which is kind of cool) with a stick. According to it, your current data is as follows (I'm cleaning up a a bit, but I'll be happy to send you how I got this: userClicks: 646, userHash: "vYfRX", userReferrers: { direct: 591, friendfeed.com: {"/": 1}, partners.bit.ly: { /sd: 3, /td: 15}, powertwitter.me: {"/": 2}, twitter.com: {"/": 24, "/home": 3, "jayrosen_nyu": 4, /jayrosen_nyu/status/2103086825": 1}, "www.google.ca": {"/reader/view/": 1}, "www.netvibes.com": {"/": 1} } - Ken Kennedy

The important point there is that of 646 clicks, 591 of them are "direct". What is direct, I'm wondering? That link isn't on http://jay.40twits.com/ , correct? - Ken Kennedy

From the FAQ: What does the "Direct" traffic in Traffic Sources mean? Direct Traffic includes people clicking a bit.ly link from: * Desktop email clients like Microsoft Outlook or Apple mail * AIR applications like Tweetdeck or Twirhl * Mobile apps like Twitterific or Blackberry Mail * Chat apps like AIM * SMS/MMS messages * It also includes people who typed a bit.ly link directly into their browser - Ken Kennedy

Yeah, that one was a bit.ly only link I sent out to my Twitter feed from Marketwatch. bit.ly tells me 632 out of 722 people total were mine. - Jay Rosen

That's what I thought. - Ken Kennedy

I don't know what that means, in the larger sense; the assertion appears to be that they're all real clicks, though. With the number being so much higher than tr.im, the next thing I'd think to look at is whether or not tr.im is not counting links somehow, or aggregating things up in some subtle way. - Ken Kennedy

On second thought, I wonder if it's possible that some mobile/AIR apps (like Twirl) try to do some sort of resolution against the URL that is incorrectly "counted" as a click. You have 21,000ish followers...it doesn't seem unlikely that a certain number in the low hundreds could be online at any particular time, using Twirl or other AIR clients. That might explain a bunch of fast "clicks" as the link flowed through. (Note: this is a complete hypothetical, though I'd think it's possible to test.) - Ken Kennedy

OK, I think we're onto something. I just pushed a non-shortened url from my blog (about 10 hits/day normally. LOL) into my FF stream (which pushes to Twitter) while watching my logs...a FF hit, and a GoogleBot hit right away, but that's pretty much it. Then I bit.ly'd another url, and pushed it in, and an immediate stream of hits to my server log; already 30 in about 5 minutes. (TweetMeBot, twitturls, etc.) Now to see what happens with a tr.im URL, and also to compare my actual server hits to each site (bit.ly, tr.im) numbers. - Ken Kennedy

It was the "immediate rush" that surprised me, Ken. But then maybe it's really a delayed rush from tr.im that should not be there. - Jay Rosen

bit.ly appears to be lagging a bit, but it's numbers for that shortened url (bit.ly/MXEMo) currently show 6, all "direct". Still rising. Will watch. Setting up tr.im url now. - Ken Kennedy

Zoom! tr.im going strong as well. Very interesting. I note that trim stats do try to guess what's a bot, but that's pretty iffy. They count 30 visits in last couple of minutes (hm...very similar to bit.ly number), but then break that into 17 humans and 13 bots. That being said, I'd be VERY surprised if 17 people have already looked at that. - Ken Kennedy

Oops...the 30 I mentioned is NOT similar to bit.ly number, but is close to actual server log hits. - Ken Kennedy

Jay (or Dave...whomever tr.im account it is)...login to tr.im and check the stats directly for one of the top40 tr.ims (this link will work if you're logged in... http://tr.im/statistics/nK3P). How does the number at top40 site compare to the stats #s. Is it the "human" number, or the "total visits" number? - Ken Kennedy

One question, one note: 1) Has anyone asked bit.ly about this? http://twitter.com/bitly would be one easy way. 2) Twitter switched to bit.ly for default URL shortening recently -- I imagine that caused in increase in bot traffic to index bit.ly links, although I don't have a better explanation than that. - Ryan Sholin

Asking bit.ly is good, certainly...but nothing wrong w/ having raw data to inform discussion. I don't think there's any sense that they're doing anything "wrong"...it's just the metrics seem to be different. - Ken Kennedy

I'm out for a bit, but I'll bundle up the data I've gotten for a bit later. - Ken Kennedy

Ken - the tr.im api that dave is using returns total visits. I haven't found a way to split out the bot numbers in the response. - Jeremy Felt

Interesting! I wouldn't have guessed that. - Ken Kennedy

Okay, someone is going to summarise and blog this for us, right? Bit.ly went kapluey (a tech term) y'day for a while but I always find that the numbers are about 3 to 5x higher than visitor numbers measured through google analytics or elsewhere. - WoH: Professor MOTHRA

Ken: I'm logged into tr.im but the only data I can access at the web portal is old urls I created by hand through the web interface. And the link you gave me above gives me an error message when I click it. - Jay Rosen

Jay: I'm guessing this means that Dave is having you use a separate login for Tr.im through his application than the one you used to use. - Jeremy Felt

I'm guessing the URLs are being generated by a different account then, Jay. - Ken Kennedy

Gotcha, Eric...thanks! This is interesting stuff. Yep, wrt bit.ly we're looking (at least right now) at userClicks for a specific hash. - Ken Kennedy

Okay I have an account Dave created for me with a different log-in and I guess I have been logged into that. Clicking on the link Ken gave me, I find 885 total. Humans: 402. Bots: 283. - Jay Rosen

That's probably it, Jay. - Ken Kennedy

Eric - Does tr.im group clicks by IP in a certain time frame at all or does it report only raw click numbers? - Jeremy Felt

To confirm, I did create a separate account for the bookmarklet. Jay was the second user after me, so I did it by hand. Later in the process the tr.im folk gave me access to an internal API for creating accounts so it's automated. I sent Jay the credentials for his account via email. - Dave Winer

Eric/tr.im folk (I like that)...so the way I'm reading the statistics page, I should "expect" (I realize these numbers aren't perfect) the "total visits" metric for a given link to be potentially higher than the actual hits in my server log. Your guess for the "bots" number are those services you know only want to unpack the url, etc. The "human" number should be closer to the "correct" value. Is that a fair (albeit jumbled *grin*) statement? - Ken Kennedy

Ah, HEAD requests...good point! Anyone know if bit.ly counts HEAD requests? (that's the very definition of an apple/orange technical comparison, Jay. I would hope they wouldn't, but if they did, there's no comparison). - Ken Kennedy

Correct, Bit.ly says they stopped reporting HEAD requests back in March. http://blog.bit.ly/post... - Jeremy Felt

Gotcha. - Ken Kennedy

I have a real jayrosen_nyu post ready to Tweet, should I do a bit.ly url? - Jay Rosen

Just update on my tr.im hits (which have died off, unsurprisingly): 37 total visits, 21 marked human, 16 marked bot. I have 25 actual hits in my server logs on that page since I posted the link. - Ken Kennedy

Sure! - Ken Kennedy

Okay its.... Coral Reef journalism: to a "sunken" structure tiny facts attach. The wikipedia stub: got an equivalent in news? Listen: http://bit.ly/sJiBz Tweeted: http://twitter.com/jayrose... - Jay Rosen

Okay the bit.ly "initial surge" count is 168 clicks, that's about three times more than an item would debut at at 40twits (tr.im) Typical debut count is 60 to 70. Initial count of 100 indicates an exceptionally fast link at http://jay.40twits.com/. - Jay Rosen

You can see the "surge" at the info page. http://bit.ly/info/sJiBz - Ken Kennedy

Good link for example, too...b/c according to the API, you currently have all the clicks for it ("clicks": 341, "userClicks": 341, "userHash": "sJiBz"). So the "total" clicks equal your userClicks...there's no other bit.ly url for this that's been clicked on. (yet). Of 341, direct is 331, FF 1, partners.bit.ly 4, powertwitter.me 1, twitter.com 4. - Ken Kennedy

Minute by minute hits, first few minutes (this'll fall off the graph at bit.ly eventually): 1, 62, 81, 65 55, 35, 24, 18, 3, 4, 5, 7, 2. I don't think that's people, you know? - Ken Kennedy

Oh, I hear what you're saying, Eric. And really, that ratio doesn't even matter for purposes of what Jay's trying to determine...why the bit.ly numbers and the tr.im ones are so different. But your comment about people processing urls is a good point: it may be that some tools looking at twitter are only processing some shortener namespaces, but not others. That would explain a lot. - Ken Kennedy

Okay, so what do we know as a result of these investigations? - Jay Rosen

For click data, unless you have your own service and can see the whole picture, use 1 tool to gauge the popularity of clicks on a pure trends basis alone, not by numbers. - Jeremy Felt

Gotcha, anything else? - Jay Rosen

I'd say that it's interesting to note that the "initial surge" should probably be completely discounted in terms of interest...I assert that in the vast majority of cases, it's basically all automated stuff. That being said, good stories still "rise to the top"...the clicks over time add up. Having two services with different "initial surges" makes comparison difficult (impossible, really), as Jeremy noted. - Ken Kennedy

My gut also tells me that we've learned that there are tiers of shorteners...I think some get more attention paid to them by the automated tools than others. bit.ly's initial surge is larger because it (like Twitter in the "what are you doing" space) gets more attention from the people making tools to analyze the data. Not a super surprising piece of info, but worth thinking about. - Ken Kennedy

And also, those numbers have only a tenuous relationship to actual web hits to the page. At this point, my story I pushed through bit.ly has 9 bit.ly hits (though I know I did a couple of those myself accidentally), and 18 page hits (since I pushed out the link). My tr.im url has 37 total visits, 21 of which are deemed to be "human" by tr.im. The page has 25 hits (since I pushed out the link). - Ken Kennedy

And of the 25 page hits on the story linked to by tr.im, I can tell several of them are bots by the user agent (Googlebot, TweetmeBot, etc.) Has ANYONE actually read that story? I'll try to determine that (based on other GETs in logs around the hits), but I'd honestly be surprised. - Ken Kennedy

Okay, so... it seems the little url shortener mystery was isolated to some kind of auto-counting by services that have targeted some url shorteners but not others, and that the critical piece of evidence is the "rush" in the first few moments, and so the "number clicked" meter is only good for trends over time. This is all based on what we can know without further digging or information from the firms. Best practice: adopt one shortener; stick with it, watch for updates on the issue. Does that capture? - Jay Rosen

Sounds right to me. Trends, not numbers. :) - Jeremy Felt

Exit question: why did 12 people tag this as one of their likes on FF? - Jay Rosen

1) contains useful information, 2) a fascinating example of the worth of social networking - Bora Zivkovic

Does Bit.ly subtract bots automagically? - Kathy E Gill

Yep, sounds right to me, Jay. As for liking it, my interest is obvious, but I totally agree w/ Bora; in general, it has useful info ("liking" is often like bookmarking for me...I can search and easily find it again with that hint), and it was also a great discussion/example of why FF is so interesting. - Ken Kennedy

continuing Bora and Ken's answers to exit question: 3) to make it easier to find in future (search by likes) 4) to publish it to likes page (like retweeting except by reference instead of by value/copying) 5) to send Jay some love/goodwill - carl morris

Well, I should definitely add: thanks, everyone, particularly Ken and Jeremy and Eric for stopping in. Thanks for your kind assistance. - Jay Rosen

I read the whole thread and "liked" it because 1) it was started by someone with a sound reputation and a history of investigation, 2) the interaction of journalist and geek was interesting, and 3) the issue of what data can we trust in Twitter-land is important (especially with the growth of URL shorteners). Thanks to all who contributed. - Chip Roberson

I read the whole thread and "liked" it because I have clients who are starting to ask for some ROI with their Twitter usage. The see numbers and want them bigger. I want to be able to give them perspective. - Kawika Holbrook

Some of my clients like the tr.im stats page because it's easy, fast, and separates humans from bots. Others like bit.ly stats page because it includes comments from the page itself as well as Twitter and Friendfeed comments. What they really want is one tool to see at a glance what's popular, where the conversations are happening, and who/what carries the most Google juice. - Kawika Holbrook

My clients wouldn't care about this discussion, but I do because it helps me make a recommendation based on facts, not guesses. I would hope Twitter's use of bit.ly as the default shortener doesn't ultimately swamp all others, if only to ensure continued innovation. - Kawika Holbrook

I actually noticed this problem a few months back when rolling up aggregate statistics for the @nytimes account. I contacted Bitly to ask them (kinda surprised nobody else has, pretty easy to do). Upshot is bitly counts expansions of the URL, not clickthroughs. So, my twitter client autoexpands bitly links and that counts for a hit, even if I don't click it. They filter out bots and such, but plugins like that are still hits. - Jacob Harris

Of course, any redirection service that counted only hits could still encounter situations where the redirect is not followed (and they don't really have any way to know that), so there will always be some overcounting. And of course, most web analytics is based on Javascript on the page which might not be executed, so there might also be some undercounting. But bitly was giving me spikes of 4-5x my web analytics totals from WebTrends. - Jacob Harris

Here is the last message I received from Bitly asking when I noticed a decline in the numbers: - Jacob Harris

Bit.ly is still counting decodes and not necessarily only actual human clickthroughs. We are not counting: - decodes that we identify as bots based on their user-agent strings, and we are working on more advanced techniques for bot identification. - HTTP HEAD requests, which some third-party tools use to get meta-information (such as the destination link) without fetching the entire response body. - calls to the /expand method made available by the API (docs are here), that we encourage developers to use when building applications. Other requests to bit.ly links that are made by humans, browser tools, or other agents are not distinguishable from actual human clickthroughs, and consequently we count them. - Jacob Harris

It seems like the general problem is there are 2 ways you can expand a link via bitly. One of which is to do a GET request and get the redirect (that is what a user would do and should be counted for them, but it's a problem when bots do it). Apps should be using the separate expansion API that doesn't count that as a hit, but the problem seems to be that many programs and plugins are still doing a GET on the short URL to expand (and being counted). - Jacob Harris

Essentially, ALL URL shorteners have this problem, but Bitly is probably exaggerated because it is so prominent in usage compared to the rest. I apologize for not blogging about this (I did tweet about it at a few points), but bitly support (that contact link on the bottom) has been very helpful in explaining the issue. - Jacob Harris