Comments on: The problem with measurement http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/ by Jeff Jarvis Tue, 07 Oct 2008 04:50:22 +0000 http://wordpress.org/?v=2.5.1 By: Metrics in Conversational and Community Marketing http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-383097 Metrics in Conversational and Community Marketing Sat, 20 Sep 2008 13:01:23 +0000 http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-383097 [...] what can be done? Jeff Jarvis points out that the problem lies with measurement. I agree, as there is only value in a system where all of the people involved agree on what the [...] [...] what can be done? Jeff Jarvis points out that the problem lies with measurement. I agree, as there is only value in a system where all of the people involved agree on what the [...]

]]> By: Katie Paine http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-357210 Katie Paine Thu, 16 Aug 2007 20:24:22 +0000 http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-357210 You were invited to a measurement summit and haven't yet responded, so it's happening anyway. I think your readers should know that there IS a measurement summit happening in Portsmouth, NH October 4th, 2007. http://www.instituteforpr.org/education/som_program_content You were invited to a measurement summit and haven’t yet responded, so it’s happening anyway. I think your readers should know that there IS a measurement summit happening in Portsmouth, NH October 4th, 2007. http://www.instituteforpr.org/education/som_program_content

]]> By: The social media revolution (in 15 minutes) http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354983 The social media revolution (in 15 minutes) Mon, 23 Jul 2007 12:05:41 +0000 http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354983 [...] BuzzMachine » Blog Archive » The problem with measurement [...] [...] BuzzMachine » Blog Archive » The problem with measurement [...]

]]> By: carlo beccaria - blog / links for 2007-07-20 http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354867 carlo beccaria - blog / links for 2007-07-20 Fri, 20 Jul 2007 07:19:13 +0000 http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354867 [...] BuzzMachine » Blog Archive » The problem with measurement (tags: metrics) [...] [...] BuzzMachine » Blog Archive » The problem with measurement (tags: metrics) [...]

]]> By: Is Link Authority Dead (Dying)? - Internet Insider Report http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354853 Is Link Authority Dead (Dying)? - Internet Insider Report Thu, 19 Jul 2007 19:43:22 +0000 http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354853 [...] Jeff Jarvis notes Nielsen//NetRatings’ announcement that they would no longer measure page views, and instead would measure the amount of time users spend at a site. While that may be a part of the next equation, Jarvis thinks that won’t be good enough in an age of instant messaging and other widgets that are always on, but not always used. [...] [...] Jeff Jarvis notes Nielsen//NetRatings’ announcement that they would no longer measure page views, and instead would measure the amount of time users spend at a site. While that may be a part of the next equation, Jarvis thinks that won’t be good enough in an age of instant messaging and other widgets that are always on, but not always used. [...]

]]> By: terrainnova.org http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354798 terrainnova.org Wed, 18 Jul 2007 12:54:22 +0000 http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354798 <strong>Time for a new PageRank?...</strong> I’ve been trying to follow the recent (i.e. today’s) discussion about how we need a new measure of ‘popularity’ (in the broadest of senses) other than Google’s PageRank concept. PageRank is based on the links pointing from... Time for a new PageRank?…

I’ve been trying to follow the recent (i.e. today’s) discussion about how we need a new measure of ‘popularity’ (in the broadest of senses) other than Google’s PageRank concept. PageRank is based on the links pointing from…

]]> By: Social media index « Technobabble 2.0 http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354770 Social media index « Technobabble 2.0 Tue, 17 Jul 2007 12:56:50 +0000 http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354770 [...] a moment too soon. - Hugh Macleod Review of the new influencers - Drew B’s take on tech PR The problem with measurement - BuzzMachine by Jeff [...] [...] a moment too soon. - Hugh Macleod Review of the new influencers - Drew B’s take on tech PR The problem with measurement - BuzzMachine by Jeff [...]

]]> By: The Interactive » New media needs new measurement http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354634 The Interactive » New media needs new measurement Fri, 13 Jul 2007 11:22:24 +0000 http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354634 [...] We can debate whether the length of time visitors spend on a website (the new metric employed by Nielsen) is a better reflection of a site’s success than page views, but Jeff Jarvis gets it right when he asserts the important questions being asked should not be about how we improve exisiting measurement tools, but “about what’s not being measured“. [...] [...] We can debate whether the length of time visitors spend on a website (the new metric employed by Nielsen) is a better reflection of a site’s success than page views, but Jeff Jarvis gets it right when he asserts the important questions being asked should not be about how we improve exisiting measurement tools, but “about what’s not being measured“. [...]

]]> By: David http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354617 David Fri, 13 Jul 2007 00:10:31 +0000 http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354617 Jeff, Sorry for the tone in my previous reply - I was disappointed that you didn't mention some of the people who have been trying to bridge the (admittedly) vast gaps that exist in the measurement field and by your pessimism, but perhaps this is just another testament to a lack of success to date. FWIW you'll probably find that we don't disagree much if at all. Anyway, there's obviously a tremendous disconnect between the brand advertisers/big agencies and the internet ad community. Part of the problem IMO is that agency media buyers simply don't understand the medium - they are fixated on the equivalent of TV reach and frequency metrics that Nielsen NetRatings and ComScore provide (the page view metric is/was really more of a--flawed--proxy for available ad inventory). At least as far as I can tell, they don't seem to realize that these numbers may not have much of anything to do with how many impressions their banner ads receive. Maybe it's an inability or unwillingness to recognize that the web is not--and will never be--a mass medium? I don't really know, but if you're a big agency with massive infrastructure and overhead--much of which can eventually be replaced by computers--you probably don't want to consider the possibility that, with the possible exception of outdoor, etc., the future might not look very bright for effectively reaching mass audiences. Nice (if long) summary here - http://blog.jimnovo.com/2007/06/12/media-measurement/#behave You're right that the IAB has been (very) behind, particularly given how long it took them to address the disparities between the panel-based measurement services--specifically Nielsen NetRatings and ComScore--and internal server logs. Inevitably one or the other will mention the effects of cookie deletions (which can vary widely from site to site) and non-US visitors, but I also think as you do that there are methodological problems, which are probably too numerous to go into here. As you wrote in your original post, it's really not clear whether a sample panel, regardless of size, can produce accurate projections when the audiences are so fragmented. It's difficult enough for small websites, let alone RSS feeds, widgets, and facebook applications. None of these major measurement firms IMO have shown that they are capable of moving proactively to address the emergence of these technologies. Incidentally I think Facebook has opened up a new world in terms of identifying influencers because application developers could establish betweenness and other centrality measures based on how a given application (or some social function present within it) is adopted by other nodes across multiple networks. Jeff,

Sorry for the tone in my previous reply - I was disappointed that you didn’t mention some of the people who have been trying to bridge the (admittedly) vast gaps that exist in the measurement field and by your pessimism, but perhaps this is just another testament to a lack of success to date. FWIW you’ll probably find that we don’t disagree much if at all.

Anyway, there’s obviously a tremendous disconnect between the brand advertisers/big agencies and the internet ad community. Part of the problem IMO is that agency media buyers simply don’t understand the medium - they are fixated on the equivalent of TV reach and frequency metrics that Nielsen NetRatings and ComScore provide (the page view metric is/was really more of a–flawed–proxy for available ad inventory). At least as far as I can tell, they don’t seem to realize that these numbers may not have much of anything to do with how many impressions their banner ads receive. Maybe it’s an inability or unwillingness to recognize that the web is not–and will never be–a mass medium? I don’t really know, but if you’re a big agency with massive infrastructure and overhead–much of which can eventually be replaced by computers–you probably don’t want to consider the possibility that, with the possible exception of outdoor, etc., the future might not look very bright for effectively reaching mass audiences.

Nice (if long) summary here -
http://blog.jimnovo.com/2007/06/12/media-measurement/#behave

You’re right that the IAB has been (very) behind, particularly given how long it took them to address the disparities between the panel-based measurement services–specifically Nielsen NetRatings and ComScore–and internal server logs. Inevitably one or the other will mention the effects of cookie deletions (which can vary widely from site to site) and non-US visitors, but I also think as you do that there are methodological problems, which are probably too numerous to go into here.

As you wrote in your original post, it’s really not clear whether a sample panel, regardless of size, can produce accurate projections when the audiences are so fragmented. It’s difficult enough for small websites, let alone RSS feeds, widgets, and facebook applications. None of these major measurement firms IMO have shown that they are capable of moving proactively to address the emergence of these technologies.

Incidentally I think Facebook has opened up a new world in terms of identifying influencers because application developers could establish betweenness and other centrality measures based on how a given application (or some social function present within it) is adopted by other nodes across multiple networks.

]]> By: Jeff Jarvis http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354604 Jeff Jarvis Thu, 12 Jul 2007 20:23:47 +0000 http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354604 Well, David, The IAB has been behind, too. RSS doesn't enable cookies and clickthrough measurement without redirects. Downloads don't ping back to verify viewership. P2P downloads likewise. Until recently, with podcasts, neither allowed dynamic and targeted serving of ads. And there is so much more to measure, like who starts memes and their biorhythms. I've researched plenty, every time an advertiser has whined that what he wants measured isn't measured and so he won't spend him money; he likes going to upfront parties instead. We can measure much better. Wouldn't you agree to that? And why don't we have a civilized discussion about this. Unless you prefer the nya-nya-nya tone. Up to you. Well, David,
The IAB has been behind, too.
RSS doesn’t enable cookies and clickthrough measurement without redirects.
Downloads don’t ping back to verify viewership.
P2P downloads likewise.
Until recently, with podcasts, neither allowed dynamic and targeted serving of ads.
And there is so much more to measure, like who starts memes and their biorhythms.
I’ve researched plenty, every time an advertiser has whined that what he wants measured isn’t measured and so he won’t spend him money; he likes going to upfront parties instead.
We can measure much better. Wouldn’t you agree to that?
And why don’t we have a civilized discussion about this. Unless you prefer the nya-nya-nya tone. Up to you.

]]> By: David http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354600 David Thu, 12 Jul 2007 17:45:37 +0000 http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354600 Jeff, There have been web measurement conferences for years, most notably the Emetrics Summit. There are two bodies responsible for developing web metrics standards: the Interactive Advertising Bureau (IAB) and the Web Analytics Association (WAA). Both have been around for many years. There are plenty of ways to measure RSS, widgets, P2P downloads, and every other means of content distribution - any server call can theoretically be tracked. The tricky part is integrating the data collected for each of these distribution models and determining their relative value for both content producers and advertisers. Additionally I would say that there are many others who are far more qualified than you to speak with authority on this subject. I would recommend doing some research before you post anything further on this subject. Jeff,

There have been web measurement conferences for years, most notably the Emetrics Summit. There are two bodies responsible for developing web metrics standards: the Interactive Advertising Bureau (IAB) and the Web Analytics Association (WAA). Both have been around for many years.

There are plenty of ways to measure RSS, widgets, P2P downloads, and every other means of content distribution - any server call can theoretically be tracked. The tricky part is integrating the data collected for each of these distribution models and determining their relative value for both content producers and advertisers.

Additionally I would say that there are many others who are far more qualified than you to speak with authority on this subject. I would recommend doing some research before you post anything further on this subject.

]]> By: Cooler Heads http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354592 Cooler Heads Thu, 12 Jul 2007 12:27:15 +0000 http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354592 I specialize in measurement of things like behavior (difficult to be sure). Yes, you should have a measurement. And you should respect the power of measurement rather than seeing it as some nasty stepchild that you can't quite shake out of your house. The webworld is full of smart people, and someone needs some kind of algorithm to capture--as well as possible--the real behavior of people reading blogs, etc. My suggestion, have the summit. But first ask a couple of really smart measurement types to do some qualitative research on web behavior so that the summiteers can begin to form items for large-scale measurement. That's what advertisers want, large-scale measurement. I think this is fascinating, and whoever comes up with a good method could make a lot of money. I specialize in measurement of things like behavior (difficult to be sure). Yes, you should have a measurement. And you should respect the power of measurement rather than seeing it as some nasty stepchild that you can’t quite shake out of your house.

The webworld is full of smart people, and someone needs some kind of algorithm to capture–as well as possible–the real behavior of people reading blogs, etc. My suggestion, have the summit. But first ask a couple of really smart measurement types to do some qualitative research on web behavior so that the summiteers can begin to form items for large-scale measurement. That’s what advertisers want, large-scale measurement.

I think this is fascinating, and whoever comes up with a good method could make a lot of money.

]]> By: ethority blog » Messbarkeit im Netz http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354590 ethority blog » Messbarkeit im Netz Thu, 12 Jul 2007 12:00:09 +0000 http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354590 [...] Buzzmachine hat man sich mit der Frage auseinandergesetzt, wie man Messwerte im Netz bewerten, bzw. welche man [...] [...] Buzzmachine hat man sich mit der Frage auseinandergesetzt, wie man Messwerte im Netz bewerten, bzw. welche man [...]

]]> By: Jacob http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354570 Jacob Wed, 11 Jul 2007 21:39:57 +0000 http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354570 Jon Lund, director of the Danish Media Online Industry, had an interesting blogpost on this issue yesterday (http://newmediatrends.fdim.dk/2007/07/audience-measurement-giant-leap-forward-now-were-in-the-race.html). Apparently, they combine the raw statistics with monthly surveys by a polling institute to get qualitative data too. Jon Lund, director of the Danish Media Online Industry, had an interesting blogpost on this issue yesterday (http://newmediatrends.fdim.dk/2007/07/audience-measurement-giant-leap-forward-now-were-in-the-race.html). Apparently, they combine the raw statistics with monthly surveys by a polling institute to get qualitative data too.

]]> By: Greg0658 http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354564 Greg0658 Wed, 11 Jul 2007 19:28:46 +0000 http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354564 Easy. Labor Based Society. Get off the monetary base. Get up, go to work, get off work, then eat drink and be merry. Easy. Labor Based Society. Get off the monetary base.

Get up, go to work, get off work, then eat drink and be merry.

]]> By: Tansley - addendum http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354563 Tansley - addendum Wed, 11 Jul 2007 19:13:52 +0000 http://www.buzzmachine.com/2007/07/11/the-problem-with-measurement/#comment-354563 Truly interesting topic, Jeff. The problem isn't new: television has had the same problem for decades, particularly since the advent of the remote - Neilsen reflects viewing numbers, but not who is clicking the mute button for a kitchen trip during commercial breaks, and who isn't. Click-thoughs only measure directed interest, not exposure to memes or names. Page views don't take into account pop-up blockers. I once used a firewall (Agnitum) that literally blocked ALL flash from my surfing. (No, I don't own stock in them, btw.) For a few weeks, my surfing was pristine, quiet, quick-loading and virtually ad-free....BUT (and this was a big one) it also blocked a lot of java scripting...and I literally found myself unable to get into some sites I WANTED to visit, since they relied on java triggers. So...I uninstalled Agnitum and went back to McAfee. Live and learn. And no, TIME is really no adequate measure, either. I can spend hours at a blog and never notice the marginals. It will be interesting to see WHAT will prove an adequate unit of measurement to entice advertisers. Of course, there's always response-polling (how did you learn about our site/product/service? - choose one of the following...) but a lot of people never bother with that. Keep us posted on this one. My paper is just getting into the web in a big way as we speak... Truly interesting topic, Jeff. The problem isn’t new: television has had the same problem for decades, particularly since the advent of the remote - Neilsen reflects viewing numbers, but not who is clicking the mute button for a kitchen trip during commercial breaks, and who isn’t.

Click-thoughs only measure directed interest, not exposure to memes or names. Page views don’t take into account pop-up blockers. I once used a firewall (Agnitum) that literally blocked ALL flash from my surfing. (No, I don’t own stock in them, btw.) For a few weeks, my surfing was pristine, quiet, quick-loading and virtually ad-free….BUT (and this was a big one) it also blocked a lot of java scripting…and I literally found myself unable to get into some sites I WANTED to visit, since they relied on java triggers. So…I uninstalled Agnitum and went back to McAfee. Live and learn.

And no, TIME is really no adequate measure, either. I can spend hours at a blog and never notice the marginals.

It will be interesting to see WHAT will prove an adequate unit of measurement to entice advertisers. Of course, there’s always response-polling (how did you learn about our site/product/service? - choose one of the following…) but a lot of people never bother with that.

Keep us posted on this one. My paper is just getting into the web in a big way as we speak…

]]>