Web Site Statistics

[SJMorris]

I'm pretty sure I've seen the info I want before, but I can't find it now.

Do I have any control over when the web site statistics get updated? My site recently got added to yahoo (thanks to Deb) and I'm curious if I'm getting any hits from there, but when I check the web site stats, it only gives me the stats up to a certain date. Presently, that date is sometime in the morning yesterday, which is close enough, but some days the stats will be a few days old.

Its no huge deal, but once in a while when I get added to a new site, I like to see if that site is getting me any hits, so I would like to be able to force an update of the web stats.

By the way, congrats on the marriage, FutureQuest!

- Steven

[Deb]

Hi Steven,

The "Few Days Old" was due to Andrew re-writing the whole thing... reference this thread http://www.aota.net/ubb/Forum4/HTML/000045.html for more info on what that was about....

As far as the stats go now.. they are run every night/early morning 365 days a year.

The stats are usually started between 2 and 5am.... those for yesterday were ran this morning.. stats for today will run around the same time in the morning tomorrow... e.g. once a day....

So by morning hours tomorrow you will be able to see the full stats for today..

Kinda make sense? Easy in concept hard to explain *g*

Deb

[This message has been edited by Deb (edited 01-05-99).]

[SJMorris]

Hmmm... there's still something I don't get. I'm trying to look at the "Top Referrer" to see if I'm gettin hits from certain places. The message says that it is analyzing 389 hits, but the "Daily hits" page says I have around 1300 hits per day. Just what segment of those hits is the "Top Referrer" analyzing? It seems to me like "Top Referrer" should gather statistics from some period of time (day/week/month), but I don't see that info anywhere.

- Steven

[hearts]

i am not gonna pretend to understand your question.. but i poked around there.. and in that navigation window there is the word "list" .. because of your question I went and checked mine out since i have never been there before.. and that "list" shows a print out..
well.. i hope this helped. I am just learning about this too. http://www.aota.net/ubb/smile.gif

[SJMorris]

I see something that says "More details" and something that says "All referers", but nothing that says "List".

All I'm trying to say is that I don't understand over what period of time the REFERRER statistics are taken. It doesn't look like the statistics since the beginning of the year... it doesn't even look like a full day's worth. I probably am just misunderstanding something, but I'm in that sad state where you don't even know what you're misunderstanding. Sadly, I pretty much LIVE in that state. I should get some address labels made out... http://www.aota.net/ubb/wink.gif

- Steven

[Terra]

The stats are compiled nightly and I process the entire month during each STATS run...

That is why the time taken to compile the STATS, get's progressivly longer as the month goes on...

All of the 'ALL' and 'TOP' buttons are from the first of the month, till the previous day...

The Daily/Hourly/Month are breakdowns respectivly covering that particular time period...

More Details, would take you to the 'Overall Monthly' statistics showing everything for that month...

I hope this makes sense... Stats are not always easy to interpret...

--
Terra
--Lives in Purgatory with STATS coding/processing--
FutureQuest.net

PS: STATS are not on a set time run every night -- I start them when system load allows, and depending how far in the month can take up to 6 - 8 hours to compile... It's a massive (mind-boggling) nightly job, and I'm constantly working to improve this, but please remember that my work, in no way, alters or misrepresents the final STATS outcome that you see... My work is focused on feeding 'http-analye' the raw logs, which I rotate, compress, and reverse resolve the IP's to Hostnames every night for each domain...

[SJMorris]

Thanks, Terra. Sounds pretty nasty.

My final question (believe it or not, I actually did go to the analyze page and figure this out... but the info there is geared toward running the program, not interpreting its output): why does the hit count for the 'top referrer' page seem so much lower than the 'daily' hit count?

- Steven

[Deb]

Hey Steven,

The hit count for the Top Referrer is counting the hits from that location. The daily hit count is counting the hits from that entire day which explains the difference in numbers.

I'm horrible with documention but most of the results are defined here http://www.netstore.de/Supply/http-analyze/results.html If you scroll down a bit you will find them mapped out http://www.aota.net/ubb/smile.gif

Hope this helps
Deb

[SJMorris]

Like I said, its not really important, so I don't want to waste your time... only answer if it pleases you, but this still doesn't make sense to me.

The referrer hits INCLUDES the internal hits from my own site. At least there is a listing there that IS my site and it says how many hits I had from there. So, if I add up all the hits internally, and all the hits externally, it SHOULD add up to somewhere near the same total as the number of hits per day. PLUS, Terra says the Top referrers counts the hits for the whole month... so four or five days now. If I'm getting 1300 hits per day, for four days... well, thats clearly way more than the 500 hits the top referrers is showing me.

Thanks for the reference, but I've looked over it and can't find an explantion of how hits is DIFFERENT for top referrer versus daily.

Once again, this isn't really important to me, as I can get a feel for where people are coming from as it is now, but it just seems a little off somehow.

- Steven

[Deb]

The way I interpret it is....

Daily Hits -- EVERY Hit that occurs on your site that day... this includes every image that is downloaded, every page, every everything. Just hitting a single html page that has 10 images on it will calculate 11 hits (1 html file + 10 image files)

Referrer Hits - These come from links. Let's say "link.com" has a link to "you.com" on their site. If five ppl click that link it will count as five refers from that site. Now those five clicks could all be going to your "11 hit" page... the one with the 10 images on it... so in comparison.. You have Five Refer Hits and 55 Daily hits...

See what I mean here?

The one is counting how many clicks came from somewhere... the other is counting how many actual hits were on your site. This puts the difference in numbers into perspective.. it's confusing to explain but makes sense after watching it for a while http://www.aota.net/ubb/smile.gif

Deb

[meikel]

Hi,

look at the "All Referers" page of your statistics. Somewhere at the bottom is a detailed explanation of the hits.

481 hits have no referer
493 hits have a shown referer
4205 hits are self referring (www.dragonlore.com)

That makes 5179 hits which perfectly matches the number shown in all other stats.

Many download helpers (GoZilla, Netzip, etc.) don't give referer URLs, so that's why you have so many hits without referer.

I don't know why the stats don't filter hits to http://dragonlore.com when accessed without the www in front of it.

Greetings from Bonn, Germany
[nbsp][nbsp]Meikel Wber
http://www.meikel.com

[SJMorris]

ok, thanks everybody. Now, I get it. http://www.aota.net/ubb/smile.gif

- Steven

[Terra]

Meikel,

good question - with a good answer... http://www.aota.net/ubb/smile.gif

I don't know why the stats don't filter hits to http://dragonlore.com when accessed without the www in front of it.

I once pondered this as well, and I found a good use for it... Say you have 2 different segments of your website that you wanted to report on... You could code it to where 1 would use the 'domain.com' and the other 'www.domain.com' thereby giving you 1 extra layer to work with if needed...

It's up to the end user to find a particular use to this side-effect... I believe that I can smash this all into 1 by counting with 'www' and without as the same class level... Unfortunately, I didn't discover this till well after STATS were established in production usage...

In conclusion: it's a side-effect - that can either be used by the designer / or compensated for when looking at the stats results...

Any opinions??? I could probably change this for those that needed to smash'em together...

Greetings from someone whom has nightmares from STATS processing,
Terra

[meikel]

I see no benefit in mangling www and no-www together. At my site the no-www access is very low and it is just another info I can take with me. So if you ask me, just keep it as it is. The only annoying thing is that the statement "internal referers" is a bit misleading, but once you know it it's ok.

If you have nightmares from stats processing it's time to get a new stats package !!!

Greetings from Bonn, Germany
Meikel Weber
http://www.meikel.com

[Terra]

Nah, the STATS package is awesome... It's the system level 'Stagex' FQuest programming that's a nightmare...

rotating/splitting/rarping/crunching/etc is the real snag... It has to be done though, or the bulk of the logs/stats would quickly get out of control and consume every byte of hard disk space that I have - let alone the countless hours of processing time if I didn't have the 'Stagex' system in place...

The last time they got out of control, 'Stage2-1' conversion - stats fell behind for several days - and took me awhile to get everything caught back up...

The *major* kicker to all this is the RARP portion of processing... (converting IP to hostnames)... I don't do the resolving realtime in the Apache server as the overhead is way too high, so I post-process this... I am determined for FutureQuest to offer the most detailed and accurate logs, especially the CGI logging details, that I will go the extra 10 miles to make the Stagex efforts a huge success...

RARP'ing the logs is very important to many people, and to myself as well, as I like to see the hosts - instead of just the IP's... It is also what provides your detailed 'Country' info, and several other aspects of the STATS processing results...

--Terra

[This message has been edited by ccTech (edited 01-06-99).]

[Justin]

I am new here, and now that I figured out how to interpret the stats, I love them. I can find out how many download of my program came from download.com, winfiles.com, etc. I know what sites are more important to keep updated and which ones only give me a few hits, and so much more!

It's great that you guys go the extra 10 miles! I couldn't ask much more from a web presence provider. Yet another reason I am glad my ISP ticked me off enough to go find someone elso to host my site http://www.aota.net/ubb/smile.gif

My question is this: all of the work involved in doing the stats (I still don't understand most of it yet), is it 'by hand'? And if so, isn't there a way to fully automate this process? Or would it just not yield the same kick-butt results?

-Justin (that's JustIn Case, not JustIn Time - I was born 2 weeks late)



------------------
Justin Nelson, SFE Inc.
http://www.vdj.net

[meikel]

<DON'T TAKE THIS SERIOUS !!!>

No, the process is not automated.
I heard from a well informed and trustworthy source (that of course wants to stay anonymous) that someone is doing a nightly printout of the logfiles starting at 12:01 AM using an old Epson Lineprinter. Around 6 AM the printouts are ready and the first 2 miles of paper are hand counted. The moment the statistical base is large enough, the counts are calculated up to some reasonable number and placed on the stats pages. Unfortunately they are still using Windows calculator, so don't expect 100% correct results, but I heard, they want to switch to an HP emulation soon.

The country statistics and the browser statistics is generated using a lotto number shareware program with statisfactory results.

The moon phase is responsible for the ranking of the top-referers so you get www.monster.com as the main referer each time when it's full moon!

The whole process is finished around 12 PM which I believe is very fast and a great achievement.

Sorry people, but I couldn't hold that secret any longer.

Greetings from the best informed journalist on the web
[nbsp][nbsp]Meikel Weber

</DON'T TAKE THIS SERIOUS !!!>

[meikel]

Hi again (now with some real info),

The stats generated from "http-analyze" are great. Of course one can always imagine some more info, but it is one of the best packages available to do the log-analyze job.

The main problem with stats-processing is that the program needs a lot of resources and puts a good load on the server. So I believe, FutureQuest is watching that process very closely to make sure the server stays responsive to do his main job (serving websites).

FQ always chooses the best software available as you can see in the stats, this forum and all the features they offer.

Do I need to say more?

Greetings from Bonn, Germany
[nbsp][nbsp]Meikel Weber

[Terra]

My question is this: all of the work involved in doing the stats (I still don't understand most of it yet), is it 'by hand'? And if so, isn't there a way to fully automate this process? Or would it just not yield the same kick-butt results?

It is mostly all automated...

The entire FutureQuest StageX project comprises of 11 seperate programs, custom written by myself, that control 4 seperate logging/analyzing sub-system layers... All of these programs have a specific duty, and all must work in perfect harmony...

I'll *never* ever - ever take stats for granted again, after being on the server side of the fence... Eeeesshhhh http://www.aota.net/ubb/biggrin.gif

I am looking at the Future, and making sure that the design is scalable enough to handle a TeRRaByte of logs - **and** keeping the processing window within 8 hours, **without** affecting our primary mission of serving web pages... It can be done - and I'm in the process of making it a reality...

--
Terra
STATS Engineer
FutureQuest.net
--Hmmm, what do I want to be today?--