Banned from Google

[Andilinks]

http://www.andilinks.com/731.gif
At midnight EDT Thursday andilinks.com was completely dropped from the Google index without warning or notice. Apart from the money, which is substantial, I am heartbroken over this. I am not a big-time developer with dozens of sites, I just have this one, it is my baby. I'm not one given much to emotion but I admit I've shed many tears this week-end. If I've seemed quiet lately this is why.

It may be that someone that I have rubbed the wrong way (I do that from time to time as you know) has reported me for some violation real or not. I have some other possible theories but at least one SEO guy has looked over my site and could not find any reason why I should have been banned.

I in fact have been scrupulously avoiding all SEO techniques not recommended by Google itself. If anyone has any idea what the problem may be I'd be most appreciative.

I have written Google the appropriate letters. If anyone has any suggestions about that I'll be happy to share the letters I've already sent.

I need suggestions on how to build my Yahoo and MSN traffic which have never been as good as they should be but are neglected because Google has always been so good to me.

I need suggestions on how I might shred and dissect Andilinks such that it might be monetized in ways other than as a website at andilinks.com...

My sadness and shock are turning to anger and I am going to turn this around. Expressions of sympathy are welcome but what I'd most appreciate are suggestions on how I might relentlessly beat this sucker until it bleeds lemonade.


Andi

[Snarpy]

So sorry to hear this, Andi.I need suggestions on how I might shred and dissect Andilinks such that it might be monetized in ways other than as a website at andilinks.com...
AndiI don't qhite understand what you are asking here.

Snarpy

[Wassercrats]

Is it possible to call Google to expedite a reply? Whatever the problem is, you can fix it, and you'll be back.I need suggestions on how I might shred and dissect Andilinks such that it might be monetized in ways other than as a website at andilinks.com...You can create a new website with most of the same content, but changed a little, maybe under a different name, but you need to know what happened. You'll find out.

[Andilinks]

I don't qhite understand what you are asking here. The data and the skills I've used to compile and edit Andilinks are valuable to many in a variety of ways. I don't expect quick answers on this whether from others or the from cauldron that is my meager brain.

But make no mistake, this data has value and now I need to exploit it. Since the beginning of 2005 I have been running Andilinks as a business and this is the first true crisis--cash flow has been very easy and the margins very fat until now. This is a whole new mode. I am asking for suggestions but in the end I know that it is I that must do the heavy lifting.

Is it possible to call Google to expedite a reply?Until I get a reply to my first email (or realize there'll be no reply) I will hold off on this. From what I've read over at WMW and elsewhere a polite but steady stream of emails is best. Google tends to be aloof when it comes to webmasters asking about this, and especially will penalize impatience or arrogance. I want to be very careful with this.You can create a new website with most of the same content, but changed a little, maybe under a different name, but you need to know what happened. You'll find out.Yes, I've thought of this. I need to be careful about duplicate content, especially content duplicated from a banned site. I would like to sell pages outright to another website since I want to narrow the scope of Andilinks to Women's Wear, Pop music, and a few other related topics. I'd like to sell off the tech pages.

Thanks :)

Andi

[Don]

If you do a MSN search on "The links are checked frequently, details" you'll find a number of sites that have duplicated the content of a particular page on your site for the purposes of the MSN cache but then redirect to the actual page on your site when you click on the live URL. What the implications of that are I do not know.

--
Don

[Andilinks]

Hmmm... the answer may indeed lie with these but I'm baffled as to how right now.

A Google search (even now) on the term "andilinks" returns 36,000 results and recently this figure was as high as 56,000. Before I began the website that term returned two results--one personal teen site long-gone and one Dutch porn site, which still exists and remains one of those 36,000.

Many of these are so-called "scraper sites" which collect content from other sites to monetize it with Adsense. There is a theory afloat that the purge that occurred Thursday night was directed at these scrapers. Andilinks differs from a scraper site in several important ways, hand editing being one. But it is possible I was swept up in this. So it is important that I carefully point out the differences to Google...

Thanks, Don.

Andi

[Juan G]

Many of these are so-called "scraper sites" which collect content from other sites to monetize it with Adsense. There is a theory afloat that the purge that occurred Thursday night was directed at these scrapers. Andilinks differs from a scraper site in several important ways, hand editing being one.
Yes, it seems that Google is employing more than algorithms, and maybe one of the people who are working for Google to rate sites misunderstood completely your directory and the spamguide (a confidential document that seems good and fair, but Google doesn't like how it's circulating on the net).

If you do a MSN search on "The links are checked frequently, details" you'll find a number of sites that have duplicated the content of a particular page on your site for the purposes of the MSN cache but then redirect to the actual page on your site when you click on the live URL. What the implications of that are I do not know.
Perhaps Andi is a victim of 302 redirect hijacking? That's one of the possibilities indeed. (A Google search on this gives detailed info).

Is it possible to call Google to expedite a reply? Whatever the problem is, you can fix it, and you'll be back.
This seems good advice.

[Stecyk]

Hi Andi,

First, my heartfelt sorrow for losing your Google ranking.

I don't know if my information will help or not, but I thought I would pass it along.

http://www.google.com/webmasters/guidelines.html

Keep the links on a given page to a reasonable number (fewer than 100).

Good luck.

Best regards,
Kevin

[Stecyk]

Hi Andi, you've probably already seen this article:

Help! My Site Has Disappeared From Google! (http://www.searchenginepromotionhelp.com/m/articles/search-engine-problems/google-ban-1.php)

But on the off-chance you haven't, I thought I would flag it to your attention.

Best regards,
Kevin

[Wassercrats]

Your back, I think, at least in a small way. http://www.google.com/search?hl=en&lr=&safe=off&rls=GGLG%2CGGLG%3A2005-21%2CGGLG%3Aen&q=site%3Aandilinks

Oh, wait.... what site is that :umm: No page there.

[cindik]

Your back, I think, at least in a small way. http://www.google.com/search?hl=en&lr=&safe=off&rls=GGLG%2CGGLG%3A2005-21%2CGGLG%3Aen&q=site%3Aandilinks

Oh, wait.... what site is that :umm: No page there.

No tld. Strange.

[Wassercrats]

Oh, I forgot the TLD.

[Wassercrats]

Since this (http://www.google.com/search?sourceid=navclient&ie=UTF-8&rls=GGLG,GGLG:2005-21,GGLG:en&q=site%3Apolisource) returns no results, maybe Google messed up www.andilinks.com and turned it into plain www.andilinks and then couldn't find any files when they were appended to that messed up URL so they eliminated it from their directory.

[Andilinks]

Thanks everyone for the suggestions and kind words, I am now awaiting a reply from Google which if/when it arrives will supposedly explain what the problems are.

Keep the links on a given page to a reasonable number (fewer than 100).Large numbers of links per page have appeared on Andilinks from the very beginning, even before being indexed by Google. This is more a feature of the older pages and while it may have affected ranking or SE results position it certainly never was cause for a ban. My more recent pages do have fewer links per page.

I am at the moment optimizing for Yahoo, (keywords) something that will help whether I get back into Google or not. I am confident that I will eventually get back in, at least partially.Perhaps Andi is a victim of 302 redirect hijacking? Yes, I've been hearing of these for some time, though I don't entirely understand how they work or what the remedy is. Again, I await Google's reply. If it is some technical problem such as this I would think they would have some clue...

Thanks again, I'll post progress here (soon, I hope).

It is Monday 9:00 am in Mountain View and I'm sure they are swamped with emails like mine, since this does seem to be a large scale ban. So it is much too soon to be giving up hope or even worrying.

Andi

[Andilinks]

... from our AdSense team passed your email on to our User Support team so we can assist you. Unfortunately, we are unable to send personal responses to all of the requests we receive to review individual website content. Websites can fall out of our index for many reasons, including penalization.
... Certain actions such as buying or selling links to increase a site's PageRank value or cloaking - writing text in such a way that it can be seen by search engines but not by users - can result in penalization. Please review our quality guidelines at http://www.google.com/webmasters/guidelines.html for more information. If you identify problems with your website and make the changes necessary to comply with these guidelines, please do not hesitate to contact us.

We are sorry that we cannot provide individual assistance at this time.

Regards,
The Google TeamNot exactly encouraging. I will be scouring my site design since I haven't knowingly broken rules. I do use certain text characters the same color as the background for spacing. Changing these to image spacers will require a major revamp, maybe I will trim out all the low-traffic pages. Whew!

Like I said, these design elements have been present for years, all of a sudden they get me banned.

Andi

[Andilinks]

And so I must undergo a major site-redesign without revenue not even assured that what I am doing will fix the problem.

Google has definitely fallen from favor with me.

Andi

[Juan G]

Help! My Site Has Disappeared From Google! (http://www.searchenginepromotionhelp.com/m/articles/search-engine-problems/google-ban-1.php)
On this article linked by Stecyk, we can read several of the possible explanations. For instance, on page 2, linking to bad neighborhoods.

It happens that good links become bad links when some people buy domains, do something wrong and get banned by Google. Therefore, it's good to check our links from time to time. I think you do it, but there are really many links on your directory.

(This is just an example of one of the many possibilities).

[Wassercrats]

Strictly following the guidelines is a good idea anyway. It still might have helped your website if this never happened, and you should have phased it in. It sounds like if you make a change or two and contact them again, you'll get a more personalized response.

[Andilinks]

Therefore, it's good to check our links from time to time. True, but just checking for dead links has become a major chore. Checking for "bad neighborhoods" becomes prohibitive and I won't even know if the effort is paying off.

My website will become much less useful in order to get indexed. This is becoming a very huge problem, all because the Google search team "doesn't have time."

I think creating new websites as tests will be easier but with the sandbox and no revenue it also could become prohibitive.

Andi

[Andilinks]

The Andilinks website as it is now produces SOME revenue from Yahoo/MSN traffic so making wholesale changes there jeopardizes that flow with no assurance that it will satisfy Google.

Andi

[Stecyk]

Hi Andi,

You might wisht to use http://www.dead-links.com/ . This site will spyder your site looking for dead links. It only goes through a maximum of 15,000 outbound links, and you appear to have 33,000+ outbound links. Perhaps, you could set up a dummy site or something and divide your sight into manageable chunks? Anyway...this service is free.

Best regards,
Kevin

[Andilinks]

I do use the Xenu link slueth, and though it is burdensome it is probably still easier than dead-links.com because it provides me with a report that I can import directly into my database for matching and deletion.

But in any case I doubt that dead or broken links could account for the ban.

Thank you Kevin I appreciate your help.

Andi

[Stecyk]

Andi,

But in any case I doubt that dead or broken links could account for the ban.

I agree. My thoughts were that might be a culmination of factors. A few demirit points here and there, and then boom.

It would be nice if Google had a spot where you could enter a URL and it would say why a site is delisted. At least then we could all learn and ensure that we don't fall victim. I hope you achieve success and are once again listed.

Best regards,
Kevin

[Andilinks]

if Google had a spot where you could enter a URL and it would say why a site is delisted.Unfortunately that would probably be abused heavily to scam the system. Google's guessing game is designed to weed out the scammers. But yes, it sure would be nice. :)

Thanks again Kevin.

Andi

[Wassercrats]

maybe I will trim out all the low-traffic pages.It could be that your shift to pop stuff is hurting you, even though it's bringing you more traffic. That alone wouldn't get you banned though.

[Andilinks]

I will be reconstituing pages and beginning new sites with parts of Andilinks, but with the sandbox and the cost of this, etc I must be careful.

[Wassercrats]

I just pasted two random descriptions from your diamonds page into my Google toolbar and there were hundreds of results for each. Maybe you need more unique descriptions.

[Andilinks]

Maybe you need more unique descriptions.My descriptions are mostly from the "description" meta tag of the site described which may in fact be the problem. If that is the case my selective editing of sites is not enough--my site looks too much like a high-rent Google page for Google.

My pages provide (in some cases) a far superior version of what a searcher may want for a particular keyword. I don't think it is "reasoned" quite like that by the algorithm though.

My selective editing on most pages does provide value because I eliminate the trashy and worthless sites. But bots have difficulty with this subtle distiction.

The site, and particularly the Women's Wear pages, have been developing a "non-referred" following but I'm afraid have not yet reached a tipping point where they could survive without a Google listing. It is beginning to appear that there is no easy path back to where I was, I will have to reconstitute the entire effort. This will in the long run produce a far superior Andilinks but at a greater cost. Oh well.

Andi

[Andilinks]

Well, trying to "fix" the old site for Google while it still continues to produce with Yahoo and MSN could easily backfire and yield something that nobody wants.

Trying to build a new site with components of the old while trying to "fool" Google into thinking it's not me is also a bad idea.

So I'm going to build a site with the best components of the old and not hide my identity so I can recapture the former Andilinks following. I will omit as many of the possible penalty-makers as possible on the new site and remove duplicate content on the old site as I build the new. I'm not sure if directly linking to the new site from the old is a good idea. I'll see if that gets banned too and if it does I will then have at least partially identified some trouble making components.

There are a lot of complex possibilities here so I must proceed carefully, building websites as experiments can get costly...

Comments are appreciated. :)

Andi

[Juan G]

> Perhaps Andi is a victim of 302 redirect hijacking?

Yes, I've been hearing of these for some time, though I don't entirely understand how they work or what the remedy is.
As an example, here is a probable case:
Hijacked Pages Equals Google Ban (http://forums.digitalpoint.com/showthread.php?t=22267)

[DogAndPony]

Hey, Andi...

What a pain! So sorry about this...

Do you think it's possible this is because of a human reviewer listing the site as "offensive" (in Google's parlance)?

I'm wondering if the policy of favoring "unique content" or at least "value-added" sites might have come into play.

Maybe some new reviewer who just doesn't like directory sites?

Or maybe a general Google policy shift, not just a purge of scrapers (/innocents)?

(Just thinking out loud here... I'm as much in the dark about the arcane internal machinations of Google as the next person...)

[Andilinks]

Hey Bob, thanks for the kind words and thoughts.

I am convinced that it is part of the scraper purge. My pages are hand edited, but my descriptions are hand selected exerpts from the site's own text or meta tags.

In many categories I do also add unique content to each entry as well but to my mind the most valuable editing is "editing by omission" that is, removing the useless items that typically clutter your average search page.

These distinctions make the site not only more valuable than a scraper site but much more valuable than a Google result page for the same keyword. It is a distiction that only actual users would notice--not a casual reviewer or an algorithm. Some pages/categories are better than others, of course.

I am remaking the entire site into several sites with new domain names, but still recognizeable by the following already established. I am getting over 1000 visitors a day (roughly adjusted to eliminate bots) who arrive unreferred and must be using their own bookmark file or device.

Unfortunately those who arrive using their own bookmark are unlikely to click on ads, it is the frothy (formerly 4000/day) Google users who click ads. So, it is perhaps more than just a scraper purge but a purge of sites like mine that provide competition for the organic listings on the actual Google result page. I do still get >1000/day from Yahoo/MSN and misc others but I'm afraid these may follow Google's lead in purging directories. I have already done some pleading to The Google Team with very little result. I have only gotten hand written "sorry" replies from someone who obviously didn't look very closely--or at all--at the site itself.

The 450 page Andilinks site as it stands is a direct evolution from the single page I began in January 2002 and is filled with quirks that reflect the evolution and my learning curve. Reconstituting and dividing the site will create a superior product but unfortunately one that must start anew in the rankings.

I also must be very careful not to duplicate too much from the banned site so it will be long, tedious and largely unpaid work until the new sites are rediscovered. I will identify them as "Formerly Andilinks" in a graphic.

Andi

[Andilinks]

As an example, here is a probable case:
Hijacked Pages Equals Google Ban (http://forums.digitalpoint.com/showthread.php?t=22267)I now can find only two links that redirect to my pages and I'm still unclear on how I could fix this--even if it is the problem. One of these links is a bookmark utility, I think Google should be aware of this.

Well anyway I do think it is the wholesale scraper ban that has killed my site and I have decided on the remedy I just outlined in the above post.

Thanks for the suggestion though Juan.

Andi

[Juan G]

I now can find only two links that redirect to my pages and I'm still unclear on how I could fix this--even if it is the problem.
Contact the webmasters to remove redirects or to change to normal links. Many don't know that some redirects -temporary: 302- can cause a problem (see for instance the linked case).

When the origin of a problem is unknown, it's of course advisable to fix as many different possibilities as possible.

And later, after the fixing work:

If you identify problems with your website and make the changes necessary to comply with these guidelines, please do not hesitate to contact us. (...)

Regards,
The Google Team
That's it.

[Andilinks]

...advisable to fix as many different possibilities as possible.I have decided that there are several possibilities and Google will give me no clue as to which may be the problem. Fixing some of these as a retrofit will be time/labor intensive. Since I had been planning to do a total site revamp in January anyway simply moving up the revamp will be a much more efficient way to handle this and the original site will continue to produce (albeit low level) with Yahoo and MSN et al.

I may do some optimization specific to MSN and Yahoo, but other than that old Andilinks will get no more updates.

I've decided to break the site up into several smaller sites, removing dup content and redirecting as I complete new sections. The "several site strategy" is a corollary to "eggs in a basket." I'm still unsure if such a network of sites would be hurt or helped by interlinking.

Thanks again :)

Andi

[DogAndPony]

I am convinced that it is part of the scraper purge. My pages are hand edited, but my descriptions are hand selected exerpts from the site's own text or meta tags.Yeah... I was thinking that maybe some human reviewer screwed up and didn't realize the site is human-edited.

Best of luck for a quick recovery!

[Andilinks]

some human reviewer screwed up and didn't realize the site is human-edited.It could be an algorithm to detect exerpted text, much like the copyright tool. I'm thinking of writing a script to swap out all common synonyms in the descriptions and then a quick hand-edit of those reduced to nonsense by the process. This would foil a scraper detecting algorithm. Unfortunately if it works scrapers will begin using them too. Another spam war escalation. :: sigh ::

Andi

edit: Gbot is starting to do large scale indexing again, so there may be hope. :) Or maybe just an uninformed Gbot. :(

[Wassercrats]

You can look at the website's "About" page and do something like:

Excerpts from http://www.andilinks.com/abt.htm: "Andilinks' mission is to identify products and sites that yield the best information...This is a free bookmark site and directory...It is a reflection of my interests, tastes and biases...virtually all have been reviewed by me."

[Andilinks]

Yes, clearly labeling them as exerpts would be better for human reviewers, I doubt a bot would make the distiction though. Bots are not all that good with contextual meaning or syntax. I do clearly state on each page that the "descriptions are from the site itself."

[Wassercrats]

If you combine excerpts with the meta data you've been using, Google might see that there's no single page containing all of your description and that could help. Also, since your descriptions are so short, Google might look for the entire description as-is on another website in determining whether it's a duplicate, and it wouldn't find such a duplicate if you use excerpts as in my example.

[Wassercrats]

You should report webpages like prom-dresses-discount.com/andilinks-prom-dresses/ to adsense-abuse --AT-- google.com.

[Andilinks]

Yes, but there are hundreds of them, I get 10 or 12 technorati tags a day for pages like this. Finding them all and reporting them would use up all my time. These are the actual scrapers, the worst of them--their content is worthless except to draw in searchers who then have no real option but the adsense link.

Andi

[TVB]

Yes, but there are hundreds of them,

Who submitted your url to all of them hoping for better page rank based on incoming links? When I did a search on Andilinks after you reported the problem I saw your URL all over the place on a ton of link farms. Maybe that is the problem?

[Andilinks]

...on a ton of link farms. Maybe that is the problem?These are not link farms, they are scraper sites. I have neither asked for nor given a reciprocal link since 2002. Scraper sites take search result pages and publish them, often as blogs on blogger for the free hosting. Usually they don't include links, though sometimes they do. The content of the page is usually so poor that the only coherent link is Adsense. Voila, instant Adsense spam.

Google is aware of this problem, in fact this most recent purge seems to be aimed at scraper sites though not exclusively.

Since Google simply will not tell or even give a clue on specific problems there is no way to know.

Betsy I'm surprised that you accuse me of doing so stupid a thing...

Andi

[TVB]

Hmmmm...my sites have never had that happen. Is it a directory site specific problem?

[Andilinks]

Is it a directory site specific problem?It is a keyword specific problem. Most of the scrapers that pick up Andilinks text and republish it are looking for Women's apparel and certain tech keywords... Though in general there are scraper sites for any high traffic keyword, I don't pay a lot of attention to them but certainly am aware that many of the 36,000 search returns on "Andilinks" are due to scraper sites. Reporting that many sites would take all my time, besides something as widespread as this must be already known to Google.

And I wouldn't be so sure that it's never happened to you, it is a very widespread problem.

Andi

[Wassercrats]

Sorry, the original post that was here is no longer available.

But the point was that you can search for "bodies like ours" along with a popular search term of teenage interest, and find sites that scraped TVB's site.

[TVB]

I keep a close eye on the phrase and also recognize that "bodies like ours" is a part of speech at times. I also block referrers from porn sites and non relevent sites as they become apparent to me (generally through the stats--something I have done only twice and btw, thanks for bringing another to my attention).

That said, the phrase alone brings up 13000 results and going through the first few pages indicates to me they are are legitimate sites, which is not what I encountered searching "andilinks".

Deeper in, most of the results are irrelevent and use the phrase talking about other stuff.

[TVB]

But the point was that you can search for "bodies like ours" along with a popular search term of teenage interest, and find sites that scraped TVB's site.

And I did just that with a bunch of different words and they seem to be only key word issues with no links back. I don't think grabbing keywords is a google banning event.

However, finding your url on link farms is probably another situation. A random check of what comes up with "andilinks" all gives links back to her site.

Betsy

[Andilinks]

I have never solicited links, nor do I answer emails that suggest exchanges. I do find my scraped content on a lot of sites though. If you have a good position for a high-traffic keyword there's no way to avoid this.

Betsy seems to be suggesting that I brought this on myself by placing my inbound links, to that I can only say that she is mistaken. I haven't solicited links in years, though when I start a new website I may be looking for entirely legitimate inbound links on relevant sites.

Andi

[Andilinks]

An update.

I regret to say my site has not yet been restored to the Google index.

For the first week the Googlebot only visited the site four times a day to pick up the site map. I then removed many hidden "_" characters that I have been using from the very beginning and requested a reinclusion. That was a week ago today on Aug. 5. Since that time Googlebot has been visiting my site 50-100 times a day but still no reinclusion. I received an email from them on Monday informing me that this would be forwarded to their engineers for further investigation. They also thanked me for my patience...

I do have some better idea what this is all about and will be redesigning and reconstituting the site as was my first inclination. But I couldn't just ignore the possibility that the hidden characters alone were the problem.

I do think that Google is highly inconsiderate of people like myself who are just dropped without warning or notice and are given very little guidance as to what the problem might be... I am miffed that it takes more than a week to even get a clue as to whether my first efforts have simply been a wrong guess.

Andi

[Stecyk]

Hey Andi,

I got my fingers crossed that you are soon back in Google's good graces. I think it is promising that Google Bots are roaming your site.

And thank you for sharing your adventures so that we all might be better prepared to deal with a similar outcome.

Best regards,
Kevin

[Andilinks]

Thanks Kevin,

These links may shed some light on what is happening...

http://www.washingtonpost.com/wp-dyn/content/article/2005/08/10/AR2005081001985.html

http://www.mattcutts.com/blog/step-into-my-shoes/

[Wassercrats]

I just wrote a script to syndicate some guy's Creative Commons newsfeed. The links to my version of his articles are on my homepage, and I intend to create a cron job to keep them updated. I wonder what the search engines will do about that.

[Andilinks]

At last this 56 day nightmare is over... All the old PageRanks (that I checked) have returned, I'm seeing Google referrals again--not at the old levels but that may just be a matter of time. Google is still sending referrals based on the site as it existed on July 28, even though it did spider since then. It will be interesting to see how quickly all the changes I've made since then are incorporated. After I collect my thoughts and do a few more "thank you's" I'll write a narrative of this most horrible episode and its conclusion.

Thank you to everyone who responded to this thread for your help and support.

Andi