Two people I respect tweaked me — one in a post, one in email (both with a business interest in the matter) — for what I wrote about the comScore kerfluffle (or efforts to start one). I think they misread me but that probably means I miswrote it, so let me restate to be clear:
First, I am delighted that we have the comScore research. I think this is incredibly important in the growth of blogs as a business — for those who want it to be a business — for this states their metrix in terms that advertisers will respect and understand… and buy. Remember that I pushed very hard for such research to be undertaken at Bloggercon II.
Second, the things in the results that don’t sound sensible can probably be corrected with improved methodology and I hope that comScore is smart enough to call upon the expertise of this open world to help them improve that methodology. Fred asks for Jason to hand over his server logs to help them and he asks for time and he’s right:
The issue with panel research is you need time to develop the statistical algorithms that weight the panel data correctly before you scale the numbers. And you need a very good dictionary of the domains you track. These take time to nail down. Clearly Comscore hasn’t nailed it yet, but they never said they had.
But third, even with the best study, I stand by my point that panel research — as valuable and necessary as it is for advertising to be able to judge an even playing field of properties — is at the end of the day still bullshit. Fred argues:
Jeff is so wrong about this that I find myself shaking my head on this one.First, the problem with panel research of old is small sample sizes. The panels that have been used for decades in old media are almost always less than 100,000 people primarily because it is cost prohibitive to collect data from a larger panel. Clearly that is way too small for accurate measurement.
But Comscore invented the concept of a “megapanel” in 1999 and is currently measuring over 2 million unique Internet users. That’s the beauty of the Internet, you can measure online at a scale unimaginable offline. At that kind of scale, panel research is not only accurate, its amazingly accurate. As far back as 2001, Comscore was able to predict a missed quarter for Amazon.com. And the panel size and the technology have improved significantly since then.
Well, I spent years in publishing and then the internet seeing how wrong panel research could be. People magazine allegedly had
eight readers per copy according to panel research. On the face of it, that’s absurd. But it determined the readership and thus cost per thousand and thus ad rates for People. And that paid my salary. But it was bullshit. And everybody knew it.
Online, I saw these big panel studies rate some of my smaller sites as huge and some of my biggest sites and nonexistent. And the reason for that, clearly, was that the sample didn’t have people clicking on mice in Ratbutt, Alabama. The problem is that when you measure small things — and blogs, individual blogs, are still very small — the effectiveness of a sample panel will decrease exponentially and the impact of one reader or one missing reader can be amplified to ludicrous extremes.
My fourth point is that, yes, I agree with Fred that Jason should take a deep breath and get over it and others who nitpicked the study should take it in the context of all such studies: They are all bullshit, so untie those knotted knickers and move on.
For, yes, this is an important study. It shows that blogs are big and growing. It shows the relative size of their audiences against other media properties and makes them real. It starts to give advertisers (and us) the flavor for the unmedium — our interests, our habits (e.g., political blogs get more traffic from smaller audiences), our demographics, and all that.
So what I’m really saying — though, clearly, I said it badly — is that the details and the nitpicking and the fact that it’s just bullshit doesn’t matter and so it’s not worth fighting over, especially if it can be improved. This is what advertisers need. This is what will make them buy.
My fifth and final point is that we would be foolish to stop at this kind of syndicated research as the basis of building blogs as a business, for — just like Google AdSense — it devalues and misjudges us. We have greater value in our relationships and influence and we need to find ways to measure that. No, that is not a substitute for the basic audience metrics advertisers demand. But it will prove our higher value and we can then get paid for that higher value and that is a good thing.
Jeff, there’s no such thing as getting an “A” for effort if the numbers are wrong. Bad numbers are worse than no numbers at all. There, with two sentences I’ve already overstated my case.
Calacanis has a right to be extremely pissed.
Tabulating the six blogads sellers who show up on comScore’s top 20 list (out of nearly 900 blogads sellers), Blogads are seen by at least 2.6 million readers. Which would put just those six blogads sellers at number six on the aggregated list of bloghosts. But those millions of readers and the great demographics are, in truth, beside the point. Your final plank is crucial Jeff.
Demographics are old media, and if blogs try to compete on these terms, we’re up against decades of imbedded sales relationships and cronyism… even when our numbers are brilliant. Pitching demographics works OK for now, but it’s like trying to sell cars in 1910 with the argument that they consume less hay.
Relationships, connections, influence, buzz, mavens, tipping points, networked thinking — these are the metrics on which blogs will excel old media. Old media, whether online or off, can’t follow us folk into that hyperspace.
Dear God, how the plot thickens. It used to be when somebody couldn’t figure things out they would start a revolution, go to church, something.
Today, what’s the salvation? Shop!
The lighter side of plots.
Top ten plots of all time:
Boy meets foreigners
1) Boy meets girl
2) Boy meets money, boy loses money, etc.
3) Boy meets windmill
4) Boy meets whale
5) Boy meets gun (multiples thereof; may involve aliens)
6) Boy meets The System
7) Boy meets ancestors
9) Boy meets inner whiny self
10) Girl makes casual acquaintance with inner self, realizes she prefers to shop
“Tabulating the six blogads sellers who show up on comScore’s top 20 list (out of nearly 900 blogads sellers), Blogads are seen by at least 2.6 million readers.”
I find your phrasing EXTREMELY interesting, Henry. First, as head of BlogAds, you should know EXACTLY how many ads you’re serving. Is it 2.6 million? I’m guessing not, and I’m guessing you won’t tell us the real number.
And I totally disagree with your assertion that demographics are not important. You’re not going to get the ad industry to buy based on buzz. Humans use numbers to measure things, and buzz cannot be measured. I’ll give Jeffo credit: he’s created buzz around this site, which vastly outweighs his puny numbers. Which is why he’s not a strong contender for advertising and must monetize the site in a different fashion.
I’m planning on getting a 100 pounds of yeast and some copper line.
If you make good stuff, you’ll see good numbers. What’s so difficult?
Demographics are like kicking around on a piece of ground in your hometown.
Everybody wanted to be global during the dotcom boom, because place didn’t matter. Most of them got put in their place. All politics is local. The same goes for business. Try using the same ads in Alaska that you use in Texas. Maybe you can sell eskimos ice water and Texans warm beer online.
This whole thing is a little weird. Who is doing this research? Do they have any idea what they’re doing? Hire some real statisticians for a change.
I wish I could help Jenny. I’ve been trying to follow Jeff with this numbers stuff and it keeps getting more confusing all of the time. Maybe the weirdness is just part of a gimmick to help boost the buzz and generate more numbers that nobody understands that need more explanations that create more posts. Maybe Jeff can help. I just don’t know.
When some people say measurement, what they really mean is tracking users. It’s confusing.
“Who is doing this research? Do they have any idea what they’re doing? Hire some real statisticians for a change.”
Actually, comScore is fairly respected on that level, although this particular report is marred by some bad decisions. Panels are a tool that originally came from statisticians — basically, it’s samplling.
Jeff is muddying the waters. Passalong numbers for magazines are totally irrelevent to Internet ad numbers, because Internet numbers that can be totally tracked. People choose to muddy the hard numbers. He’s actually right about many things in this post, but for some reason he chooses to piss and moan about stuff in that past that has no relation to the issue at hand, which is too bad.
Ed:
Yes, internet ad numbers can be tracked. They come off servers and out of cookies. They are countable.
This research is with a representative panel; it’s all about extrapolation from a sample. That is what makes it like magazine passalongs.
A representative panel of 2 million is good enough for 99 percent of us. It’s nothing like a magazine passalong, which I agree is pretty bogus.
The people complaining the most loudly about this — and I definitely don’t lump you in here — are the ones who refuse to share their numbers from server logs. Sitemeter doesn’t count; in my testing it’s almost as unreliable as Alexa.
What we really need is a third-party auditing system not subject to abuse. Old media never really solved that problem, and I doubt new media will, either, until they look to the C|Nets, ZDNets and the Jupitermedias, who actually solved this problem some time ago. Bloggers do like to reinvent the wheel.
Ed: Way right. We need some standards for blog sites’ collection and reporting of metrics (this is part of my screed on the need for open-source ad calls). Sitemeter as it exists today isn’t it; it doesn’t count RSS, for example. On the panels: You’re right, the large panels have gotten better, but I found even for large sites that they were at odds with what we knew to be true (at my last job) and in the case of blogs, they are measuring something much, much smaller, where the margin of error is thus much, much greater. The overall numbers are right to me. But it’s not hard at all to see how one site can be out of kilter vs another when it gets down to that level.
People do stuff online which corresponds to what they do offline. You can’t really measure that or track it. Ed’s right about reinventing the wheel.
This Comscore panel, no matter how big, is just as problematic as the old ones. If you read the methodology carefully (and god knows I might have this wrong but the *one* thing I actually have some work expertise in is primary research) they claim the use of RDD for a panel, but it’s NOT the method in which they got all the 2 million populated panel. That would be freakin’ impossible. They got the huge sample from online recruitment methods which while widley used, are no different than site pop-up surveys or people signing up for Alexa. There are more akin to marketing databases than a true research panel. The panelists they actually recruit from the RDD method get the *enumeration survey” in order to project those 2 million online recruits into a “representative sample.” I’m sure I’m over-simplifying it, or hell, maybe I have it completely wrong but their statisticians have obviously, and maybe very soundly, come up with intricate algorithms to mold those 2 million non-representative recruits into a representative sample from the enumeration survey to crunch all these numbers. I’m not gonna use the old cliche of garbage in-out because I DO generally think these numbers are pretty accurate for the overall stuff and thats what is important right now. I just question the prematurity of trying to drill down into specific blog audience numbers.
Also, can we talk about the REAL 800-pound gorilla in the room?? Jeff, you have a serious troll on your hands with that Jim Dermitt guy. Either he is genuinely dumb as bricks, or he’s mastering the biggest, most subtle con job with all these ridiculous comments.
To make sure Ed and I are on the same page: “number of ads served” was not what I commented on. Comscore suggests that 2.6 million individuals each quarter are looking at just six of the blogs we serve. That’s news to me, great news.
As Ed notes, I know how many ads we serve, but that number isn’t one that helps me convince advertisers to buy on blogs. On the other hand, I do NOT currently know something some advertisers do ask for: how many unique individuals are seeing the ads? We don’t have access to blogads sellers log files, since these bloggers are indies, hosted on multiple servers and service. Even if we did, unique user numbers from server log files often do not accurately log how many distinct individuals see a site.
I’m sure there is lots of waggle in the Comscore numbers, but the overall scale sounds right and it’s nice to have something concrete to chew on. Incumbent media trumpets every jot of data that makes them look good — bloggers should have some fun too!
The numbers of surveys are like face-to-face networking–sure, you can say it means nothing but when it comes down to it, they mean alot. When you give a potential advertiser, or the general public for that matter, some fudged math that makes you look good, you’re going to be considered credible. If you just give them the anecdotes–forget it.
Numbers act as that face-to-face meeting–they show a potential contact who you are. If they are false, it’s worse than being disingenuous and sycophantic when meeting someone–people can forget your face but they don’t necessarily forget your numbers.
Henry, first you write:
“Tabulating the six blogads sellers who show up on comScore’s top 20 list (out of nearly 900 blogads sellers), Blogads are seen by at least 2.6 million readers.”
but then you say:
“To make sure Ed and I are on the same page: “number of ads served†was not what I commented on. Comscore suggests that 2.6 million individuals each quarter are looking at just six of the blogs we serve. That’s news to me, great news.”
Which is it? Again, we all know the important number isn’t total eyeballs, but number of ads served.
?? I apologize for being dense, but I’m not getting you.