Help! Search Engine Problem (lost listings on Google)

[JRepici]

Hi,

I have a couple of pages which used to list in the top ten on Google with relevant searches, but now don't show up at all.


One of the pages is:

The Un-Methodology (http://www.creativyst.com/Doc/Articles/Mgt/UnMethod/UnMethod.htm)
This page also once produced a 4 or 5 rank in the Google task-bar, but now produces a gray bar.
The fly-over now says: No PageRank Information Available

This page is an article listed on dmoz.org. There are inlinks (based on the site-tool at Yahoo!, I have never been able to get google's inlinks feature to work).

There are two other pages with this behavior on the google toolbar, but other than to say that they were at least ON the index, I can't say where they were before I noticed this problem.


. . . . . . .

I don't know why this happened, but I noticed that there had been server problems here around the beginning of the month, based on the following stats chart:

http://www.creativyst.com/Site/stats0208.gif

Could this have anything to do with it? If not (or maybe) can anyone think of any other reasons why the above article stopped showing on the Google index?

Thanks if you can help.

-djr

[Andilinks]

Google just recklessly does as it pleases as though it were a corporation responsible only to its stockholders. I agree that their behavior is often harmful to the public interest and to the value of the Internet.

I don't think an occasional 404 will cause a loss of ranking, it would have to be a sustained absense to be dropped. I have seen PageRank persist on pages that have been missing for weeks.

[kitchin]

Google will ban your site if it thinks you are playing tricks to increase your rankings. I saw it happen to someone who hired one of those scam SEO operations. The ban lasted about four years.

[JRepici]

Hi,

AndiLinks:

Well, I understand a corporation's first moral responsibility is to its stock holders. I also understand that Google can do whatever it wants to the listing. I'm just trying to understand why. Even if the answer is "the game is rigged", one can usually find ways to shield him/herself in a rigged game by better understanding its nature.

I did find the thread on your experiences while researching this here on the forum. I'm always suspicious when people, like Google, make sweeping, easy-going statements about something as complex, deceptive, and subtle as, say, "evil".

Parenthetically: Often when people feel the need to explicitly characterize something, it's a pretty good indicator that the "something" contains little of the attribute they are trying to convince you it has. For example: "Philly" Cheese-steak, "distinguished" senator, "not-evil" corporation.



Kitchen:

I got that from my cursory research. Knowing that, I probably should have mentioned this in my first post:

1. I don't get involved in link-exchanges of any kind. I get the request e-mails, like everybody else. I used to open, and read them. They were almost always for links at sites that have nothing to do with the subject matter at my site. Sometimes they would talk about strange three- and four-way linking schemes. Since I don't have any idea why anyone would want to do that, I thought it better just to delete the e-mails when they come in.

2. I've never employed an SEO. I did some coding work for one once. It was for a Google-dance detector. Within weeks of completing the project, the client told me Google had stopped doing things that way, and that the dance-detector I had made him was no longer useful. He was quite upset, but when I offered to return his money, he would not take it ...


-djr

[kitchin]

The site that got banned did one of those things with 50 different versions of the home page, each with different keywords. I still get clients asking me about services, for SEO or whatever, that they have obviously read about in spam. Argggghhhhh!

[Andilinks]

I'm just trying to understand why.It is my opinion, based on several years of trying to figure out "why," that Google intentionally confuses the issues in order to confound those that would game the system.

I think the ongoing battle between web spammers and search (mostly Google) has made the internet a mere shadow of what it might be. But then that's just a continuation of the eternal struggle between civilization and chaos.

[JRepici]

Andilinks:

It is my opinion, based on several years of trying to figure out "why," that Google intentionally confuses the issues in order to confound those that would game the system.


It could be just the time of year here in the U.S., but when you put it that way it suggest Google and the IRS seem to have this in common. :confuz:


Kitchin:

Yeah, if I did stuff like that I wouldn't even need to wonder "why". Usually, the only advice I ever follow is what I get directly from Google. I have been, for example, using their new "nofollow" tag. Mainly to save Google from having to index a bunch of low-density/low-information-content pages from my site. Pages that nobody wants to see in their list of search results. It would really suck if they invented and recommended this technique, and then punished people for using it... But, as Andilinks points out...


-djr

[Randall]

The site that got banned did one of those things with 50 different versions of the home page, each with different keywords. I still get clients asking me about services, for SEO or whatever, that they have obviously read about in spam. Argggghhhhh! Yeah, when people ask me about that, I tell them to find a good marketing firm. Especially when the sites are local (and commonplace) by nature -- how do you expect to stand out from all the other <insert generic profession> in your own town, let alone the whole state or nationwide? Offline advertising is still important for a lot of things, unless you're Amazon. Not every site can claim to be a "destination."

I'm not as pessimistic as Andi, but I do agree with the notion that Google's primary motive is to outwit the search spammers. Unlike email, someone is actually in control of this game -- and when they change the rules, innocent people can get caught in the crossfire. :sad:

Randall

[georgeek]

....can anyone think of any other reasons why the above article stopped showing on the Google index? But it is in the index! As you can see here (http://www.google.com/search?q=un-methodology).

- George

[Andilinks]

This recent article (http://www.avivadirectory.com/downloadblog/?p=338) posted by happety (http://www.aota.net/forums/member.php?u=13640) in another FQ thread may help. There are many reference links there as well.

I'm not as pessimistic as Andi...During the two previous times that I suffered precipitous Google traffic losses I made it my full time job to figure out why. Both times the traffic returned just as suddenly as it disappeared and there was no way to know which change that I had made (if any) was responsible for the recovery. I have concluded, based on my readings of the various forums and websites that address this problem that this happens A LOT, to MANY web sites, and virtually no one knows for sure why though there are many possible causes and if you are engaged in practices meant to deceive search engines you at least have an obvious place to begin looking.

If you are not intentionally engaged in deceptive practices you should begin looking for innocent practices that may be interpreted as deceptive by Google (http://www.google.com/support/webmasters/bin/answer.py?answer=35769).

After that there is only patience. Sometimes it takes many months, sometimes less. But traffic may just return for no reason. Best of luck.

[JRepici]

Andilinks,

Thanks. I'll take a look. Now I've noticed they've removed another previously-five-ranking page from the pagerank bar.

Georgeek,

Notice though that if you try to access it by anything other than 'un-methodology' it is nowhere to be found. In fact, even if you just eliminate the dash ('-') from your search term, it will completely fall off the index. Interestingly, if you use un methodology (http://www.google.com/search?hl=en&q=un+methodology&btnG=Search) (no dash), another web site that links to the actual article is listed on the first page, but not the actual article. The actual article page can't be found, even after scrolling through the first 10 pages. When you go to the linked site listed on the first index page (http://www.nomoz.org/site/506462/the-un-methodology.html) and search for 'un methodology' (no dash) the string is not on the page or in the source.



Is this anything?

The impression I'm starting to get of google, is that its a lot like that girl in high-school who would adamantly insist that she only wanted to be with nice guys who respected her. But then, you eventually realize that the only guys getting anywhere with her are the ones who game her and tell her whatever they think she wants to hear.

In high school the moral decision every guy must face is whether or not to have integrity and treat all people with respect and dignity.

I guess I'll have to continue to look for ways to keep my self-respect, even if that means not getting into google's indexes. :)

-djr

[JRepici]

Here's something unusual...

I just tried to view my stats from past years and they are no longer there. I wanted to see if there had been a noticeable difference in how often google's bot has been accessing the site, and, well... All the old stats, --stats that have been accessible from the beginning-- are gone. Anybody else experiencing this?

-cvst

p.s. No stats, even for January of this year!
For Feb. Googlebot only made six accesses.

[Melissa]

Hi,

I've looked at other servers that my personal accounts are on and I'm not seeing any missing stats. I've also looked at one account on the same server you're on and I'm not seeing missing stats there either. However, I can't access your stats to see what you might be viewing as I don't have the username/password. Will you please send that information along to the Service Desk (http://service.futurequest.net/index.php?_a=tickets&_m=submit) so that we can take a look?

Thank you in advance,
Melissa

[JRepici]

Melissa,

Thanks. Done... with some trepidation. :)


Andilinks,

The article is great! Thanks. I only understand about 90% of what he's talking about but I'm learning. The extra links are helpful in that regard.

-djr

[Melissa]

Melissa,

Thanks. Done... with some trepidation. :)

Got it, thanks, and I can confirm that the stats on HUGO are not showing correctly. I've assigned your ticket to the sysAdmin to have a look. Please allow some additional time for investigation and follow-up.

Thanks again,
Melissa

[Melissa]

Hi again,Here's something unusual...

I just tried to view my stats from past years and they are no longer there. I wanted to see if there had been a noticeable difference in how often google's bot has been accessing the site, and, well... All the old stats, --stats that have been accessible from the beginning-- are gone. Anybody else experiencing this?

-cvst

p.s. No stats, even for January of this year!
For Feb. Googlebot only made six accesses.I'm sorry to say that the stats history on HUGO was lost when the server had to be restored back in early February:
http://www.aota.net/forums/showthread.php?postid=164849#post164849

A couple things to be aware of:
1) Any prior STATS history is lost as we don't back that up due to the enormous disk space requirements
2) Any prior web logs, same issue as #1

What we guarantee with backups is server critical files and your data files, anything ephemeral is not backed up...

[JRepici]

Sigh,

Thanks for your help Melissa. It makes sense.

Could I ask another favor? Could you check a few sites with a comparable number of pages to get some idea of what a typical number of googlebot and googlebot2 hits will normally be?

My February googlebot visits (the only month I have to compare now) produced only six hits. Going entirely on memory here, I recall there normally being a LOT more googlebot hits than six per month. Note that this is HITS, not visits.

-cvst

Also/P.S.

Just went back and checked, NO googlebot visits at all this month. Not one.

[Jarrod]

Have you tried Webmaster Tools (https://www.google.com/webmasters/tools/siteoverview) for your site.

[JRepici]

Jarrod,

Hi. Yeah, I went there after the fact but I'm not sure what to look for.

If Google thought I was practicing any of the behaviors they find offensive (e.g., linking to bad neighborhoods, buying or selling links, doing link-exchange schemes) would any of this be indicated there at the webmaster tools site? When I went there the newly-grayed-out pages simply stop existing.

Also, can you give me some idea of how many googlebot hits (0.9, 2, etc) there should be for a typical month?

Thanks Jarrod.

-djr

[sheila]

Could I ask another favor? Could you check a few sites with a comparable number of pages to get some idea of what a typical number of googlebot and googlebot2 hits will normally be?

My February googlebot visits (the only month I have to compare now) produced only six hits. Going entirely on memory here, I recall there normally being a LOT more googlebot hits than six per month. Note that this is HITS, not visits.
....

I would suggest requesting that Site Owners offer information here on their own site's stats via the forums. Perhaps provide some information on what you would consider a "comparable" site.

It does seem to me, though, that there's going to be quite a variance from site to site with Google, even with similar number of pages/files/data ... at least as I seem to think of it, Google's visits seem to depend to some extent on content, history of the site, number of inbound links, etc... (I have very little knowledge of SEO).

[Andilinks]

I have a site with 185 pages indexed online since 2002. The history is wildly varied and too complicated to get into here but I have had as many as 500 pages there. The googlebot still asks for pages I deleted over a year ago.

Currently several Googlebots visit this site, typically one does all the heavy lifting and the others are sporadic. Occasionally the busy Googlebot changes its IP# but for the period depicted below it remained the same.
Stats for visits from googlebot 66.249.70.228 only:

http://www.andistars.com/goog.gif

[Andilinks]

After posting the above chart I began to wonder if there were any significant googlebot IP ranges that I was missing since all the referrers that actually used the googlebot name that I know about are in the 66.249. range. The checks that I make are by no means exhaustive, I actually prefer to spend time researching the topic of my site and not the meanderings of wayward bots.

I have found an even greater number of hits from Google IP's which are not labeled with the "googlebot" user agent. I'm beginning to think that Google is providing a significant portion of my traffic. :)

This Google IP list (http://iplists.com/google.txt) hasn't been updated in about nine months, if anyone knows of a better list please post it. The page at the root domain gives other bot IP's.

[georgeek]

Google's visits seem to depend to some extent on content, history of the site, number of inbound links, etc...
The main variables that determine visit frequency are how frequently your content changes and your PageRank (not your toolbar pagerank which is meaningless).

After posting the above chart I began to wonder if there were any significant googlebot IP ranges that I was missing
The only way to know for sure that a bot accessing your server is Googlebot is by using a reverse DNS lookup and making sure that the name is in the googlebot.com domain and then doing a forward DNS lookup using the googlebot name. You can then eliminate the spammers and hellions who are accessing your site while claiming to be Googlebot.

- George

[Andilinks]

The only way to know for sure that a bot accessing your server is Googlebot is by using a reverse DNS lookup and making sure that the name is in the googlebot.com domain Thanks for that George but my question was the reverse: how many IP's not labeled with "googlebot" are in fact Google's bots.

The list I linked to claims that all these numbers are in fact Google, many of which visit my site without the googlebot referrer. There's no reason to doubt these numbers, just the completeness of the list.

Some not labeled "googlebot" are image bots, some are ad bots, some are labeled "wireless," and some are not labeled, perhaps they are not bots but are Google staff on their office computers.

[Andilinks]

There's especially no reason to doubt that bots in the 66.249. range are in fact Google's bots.

[georgeek]

Thanks for that George but my question was the reverse: how many IP's not labeled with "googlebot" are in fact Google's bots.
You will never know, unless of course you do what I suggested for every access to your site (which I would not recommend) and then collaborate with others doing the same thing. You can exclude these (http://ws.arin.net/whois/?queryinput=google) but I still wouldn't recommend it.

- George

[Andilinks]

I doubt that fake googlebots are skewing my stats. I do reverse DNS checks on the googlebot IPs that I actually track and only occasionally count the occurances of "googlebot" in the log as a casual cross check.

[JRepici]

Andilinks,

Wow! A big juicy steak in a sea of sizzle.

THANK YOU!


Well, I got 6 hits TOTAL for the month of Feb. and ZERO so far this month**. It looks as though Google has draped the figurative John Houseman blanket over my head here (perhaps it should be called the figurative figurative-J.H. Blanket :) ). That is disappointing in a way. It is freeing in other ways. I have been trying to come up with ways to better protect my original online content from the dregs of the Internet. This complete, totally unexplained shutdown of my site by google may give me a window to try some new things in that regard... when I get some time.


-djr

** There's one called Mediapartners-Google/2.* that produced 6 hits this month, but I don't know if that one has anything to do with Google.

[Andilinks]

Mediapartners is associated with Adwords/AdSense, do you have one of those accounts? If not the bot has probably just run astray. It hits my AdSense site a lot.

A Google search of "site:creativyst.com" shows 2750 pages indexed, it is odd that googlebot has abandoned your site entirely. I got regular visits from googlebot even when my domain was banned entirely for 54 days in 2005.

[JRepici]

Hi all,

More interesting information.

I've returned to Webmaster tools and went to a section called: Set Crawl Rate

There are three charts here. The first two are:

1. Number of Pages Crawled Per Day

and

2. Number of Kilobytes Downloaded Per Day


About a third of the way through February, these two charts drop to ZERO and STAY THERE.

The third chart is access time. It spikes right where the other two charts end, and then the trace vanishes.

The page shows the crawl rate set to Normal. And a table next to the chart says the max/avg/min crawls per a day to be 921/232/12 respectively.

You can change this, and hit a save button next to the radio selection. I'm going to try to save the existing setting to see if this will kick-start the crawl again.

Thanks for the info Andilinks. I do indeed have an account with them. Also, something I failed to mention: Their feed crawler seems to be working fine (I have maybe three or four feeds on the whole site that don't change much and it has visited nearly 100 times so far this month).

[JRepici]

Hi again,

Two new questions about this issue:

The section on Google webmaster tools at "Tools->Set Crawl Rate" has graphs of the crawls per day.

I thought these stopped in the middle of February because the horizontal month indexes mark the ENDS of the months instead of their START-DAY (I think).

1. What the graphs are now showing me is that Google IS crawling my site hundreds of times per a day. So why isn't googlebot showing in my Agents list in those numbers? I get a few a month for Feb and now for March-to-date.


2. Also, I looked for the un-methodology article in the "Links->Pages with Ex/Internal Links" sections. The URL is not there even though there ARE links to this article.

Can anyone help me with this?

-djr

[Andilinks]

My tracking of googlebot 66.249.70.220 (only) indicates 141 hits during the first six hours of today yet the log for that same period has only 48 occurrances of "googlebot." Since there are also googlebots with other IP#s I'd tend to trust the graphs. In order to check them you'd have to track all the Google IP#s that I linked in an earlier post.Also, I looked for the un-methodology article in the "Links->Pages with Ex/Internal Links" sections. The URL is not there even though there ARE links to this article.I'll check this later, I have no idea at the moment but I'll post now for the sake of #1. This question confuses me. Presumably it will become clearer when I visit the page. :)

[JRepici]

Thanks again for all your help with this Andilinks. I didn't realize that they send out un-named bots.

I've also signed up for a google webmaster help forum. Between your help and them I should be able to get to the bottom of this.

It isn't as important for me, since the site is probably a net negative when it comes to getting projects. But it is still an unexplained glitch that irritates the hell out of my personal OCD. :)

-djr