Help! Search Engine Problem (lost listings on Google) - Page 3

*Andilinks* · 03-08-2008, 04:29 PM

I have a site with 185 pages indexed online since 2002. The history is wildly varied and too complicated to get into here but I have had as many as 500 pages there. The googlebot still asks for pages I deleted over a year ago.

Currently several Googlebots visit this site, typically one does all the heavy lifting and the others are sporadic. Occasionally the busy Googlebot changes its IP# but for the period depicted below it remained the same.
Stats for visits from googlebot 66.249.70.228 only:

*Andilinks* · 03-09-2008, 08:58 AM

After posting the above chart I began to wonder if there were any significant googlebot IP ranges that I was missing since all the referrers that actually used the googlebot name that I know about are in the 66.249. range. The checks that I make are by no means exhaustive, I actually prefer to spend time researching the topic of my site and not the meanderings of wayward bots.

I have found an even greater number of hits from Google IP's which are not labeled with the "googlebot" user agent. I'm beginning to think that Google is providing a significant portion of my traffic.

This Google IP list hasn't been updated in about nine months, if anyone knows of a better list please post it. The page at the root domain gives other bot IP's.

georgeek · 03-09-2008, 02:43 PM

Quote:

Originally Posted by sheila

Google's visits seem to depend to some extent on content, history of the site, number of inbound links, etc...

The main variables that determine visit frequency are how frequently your content changes and your PageRank (not your toolbar pagerank which is meaningless).

Quote:

Originally Posted by Andilinks

After posting the above chart I began to wonder if there were any significant googlebot IP ranges that I was missing

The only way to know for sure that a bot accessing your server is Googlebot is by using a reverse DNS lookup and making sure that the name is in the googlebot.com domain and then doing a forward DNS lookup using the googlebot name. You can then eliminate the spammers and hellions who are accessing your site while claiming to be Googlebot.

- George

*Andilinks* · 03-09-2008, 02:59 PM

Quote:

The only way to know for sure that a bot accessing your server is Googlebot is by using a reverse DNS lookup and making sure that the name is in the googlebot.com domain

Thanks for that George but my question was the reverse: how many IP's not labeled with "googlebot" are in fact Google's bots.

The list I linked to claims that all these numbers are in fact Google, many of which visit my site without the googlebot referrer. There's no reason to doubt these numbers, just the completeness of the list.

Some not labeled "googlebot" are image bots, some are ad bots, some are labeled "wireless," and some are not labeled, perhaps they are not bots but are Google staff on their office computers.

*Andilinks* · 03-09-2008, 03:14 PM

There's especially no reason to doubt that bots in the 66.249. range are in fact Google's bots.

georgeek · 03-09-2008, 03:43 PM

Quote:

Originally Posted by Andilinks

Thanks for that George but my question was the reverse: how many IP's not labeled with "googlebot" are in fact Google's bots.

You will never know, unless of course you do what I suggested for every access to your site (which I would not recommend) and then collaborate with others doing the same thing. You can exclude these but I still wouldn't recommend it.

- George

*Andilinks* · 03-09-2008, 04:22 PM

I doubt that fake googlebots are skewing my stats. I do reverse DNS checks on the googlebot IPs that I actually track and only occasionally count the occurances of "googlebot" in the log as a casual cross check.

*JRepici* · 03-10-2008, 12:51 PM

Andilinks,

Wow! A big juicy steak in a sea of sizzle.

THANK YOU!

Well, I got 6 hits TOTAL for the month of Feb. and ZERO so far this month**. It looks as though Google has draped the figurative John Houseman blanket over my head here (perhaps it should be called the figurative figurative-J.H. Blanket

). That is disappointing in a way. It is freeing in other ways. I have been trying to come up with ways to better protect my original online content from the dregs of the Internet. This complete, totally unexplained shutdown of my site by google may give me a window to try some new things in that regard... when I get some time.

-djr

** There's one called Mediapartners-Google/2.* that produced 6 hits this month, but I don't know if that one has anything to do with Google.

*Andilinks* · 03-11-2008, 09:09 AM

Mediapartners is associated with Adwords/AdSense, do you have one of those accounts? If not the bot has probably just run astray. It hits my AdSense site a lot.

A Google search of "site:creativyst.com" shows 2750 pages indexed, it is odd that googlebot has abandoned your site entirely. I got regular visits from googlebot even when my domain was banned entirely for 54 days in 2005.

*JRepici* · 03-11-2008, 06:04 PM

Hi all,

More interesting information.

I've returned to Webmaster tools and went to a section called: Set Crawl Rate

There are three charts here. The first two are:

1. Number of Pages Crawled Per Day

and

2. Number of Kilobytes Downloaded Per Day

About a third of the way through February, these two charts drop to ZERO and STAY THERE.

The third chart is access time. It spikes right where the other two charts end, and then the trace vanishes.

The page shows the crawl rate set to Normal. And a table next to the chart says the max/avg/min crawls per a day to be 921/232/12 respectively.

You can change this, and hit a save button next to the radio selection. I'm going to try to save the existing setting to see if this will kick-start the crawl again.

Thanks for the info Andilinks. I do indeed have an account with them. Also, something I failed to mention: Their feed crawler seems to be working fine (I have maybe three or four feeds on the whole site that don't change much and it has visited nearly 100 times so far this month).