Weird Referer

phppete · 11-16-2003, 06:56 AM

Date: 16th, Nov 2003 11:40 am
Referer: XXXX:+++++++++++++++++++++++++++++++++++...
Entry Page: /detail.php...
Browser: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
IP Address:cpc1-cmbg3-3-0-cust9.cmbg.cable.ntl.com

Why would the above visitor have a referer as XXXX:++++++++ ?? anyone know why that is?

Also I've noticed my site is being spider by Yahoo!'s new product engine but it identifies itself as IE5.5 as below:

Browser: YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/)
IP Address:yj1002.inktomisearch.com

For those who wish to ensure SESSION ids are removed for bots/spiders and logs only contain humans (assuming you have set up your own tracking system), you should be aware of this bot 'pretending' to be a human... apparently this is Yahoo!s answer to Froogle which might suggest that Yahoo will be parting company with Google.... perhaps.

Pete

*kitchin* · 11-16-2003, 03:27 PM

In some browsers & other software, you can set the Browser (User-agent) to whatever you want. Opera, for example.

Ad for the MSIE 5.5 string, it might be so it will pass browser detection on some sites. It's just saying it can handle the same content as Mozilla 4.0 and IE 5.5. No use indexing a page that says "this site requires Netscape 4 or IE 5.5 and above, please upgrade your browser!"

phppete · 11-16-2003, 03:40 PM

My theory is that Yahoo added MSIE IE5.5 to make it more tricky to detect it as a bot and to ensure it doesn't get blocked. Many people test the user agent by looking for MSIE, or maybe I'm just paranoid... I've stopped it triggering my logging class, since it won't be whipping out its Credit Card anytime soon I don't want it in my visitor logs!

*Randall* · 11-16-2003, 04:15 PM

Quote:

My theory is that Yahoo added MSIE IE5.5 to make it more tricky to detect it as a bot and to ensure it doesn't get blocked.

If that were the case, why bother identifying the source as "YahooSeeker"? They even give the URL for their FAQ, which tells you how to block the bot.

Randall

phppete · 11-16-2003, 04:34 PM

I still stick by my theory.

*PaulKroll* · 11-16-2003, 11:57 PM

They did it for the same reason other bots still do it: because there are still a fair number of sites that give different content if they don't see at least a "Netscape 3" (compatible) user agent.

phppete · 11-17-2003, 03:23 AM

I forgot that some people still do that but these days I thought everyone designs properly. Are people using browser detection rather than object detection

!

*kitchin* · 11-17-2003, 12:59 PM

Quote:

Originally posted by phppete:
I still stick by my theory.

Then you won't want to click on that User-agent link from within your stats directory, because your browser will send a refer(r)er header, and, whoosh, Yahoo will know the address of your stats directory. Unless you've password-protected it.

phppete · 11-17-2003, 03:04 PM

Its not my stats directory, its a custom tracker for customers, I know every move they make and when

It has the usual password protection plus my own anti-hacker defences, it's hacker and script kiddie proof!

Pete

**Terra** · 11-17-2003, 03:07 PM

Quote:

it's hacker and script kiddie proof!

LOL - quoted straight from the:
"Book of Famous Last Words"...

--
Terra
--for every lock you add, there are a thousand crackers with too much free time on their hands--
FutureQuest