Using robots.txt file to increase visibility

[MTDesigns]

A while ago I had an issue with Alexa -- they insisted on listing personal information (real address, phone) that I hadn't provided to them anywhere online. They eventually removed the information, but I had a dislike for Alexa from that moment on. So, I disallowed them via my robots.txt file.

There are 2 things I noticed from doing this:

1. My page views have declined (I am not sure if the reason is from disallowing Alexa access or if I am actually suffering from my spam issue (http://www.myparentime.com/spam.shtml) back in Jan-Feb) .

2. I did not realize that disallowing Alexa would also include removing my sites from the Internet Archive. I used to think it was a good thing...having my site logged as proof of my work. But since disallowing Alexa, my site does not show up. If I allow Alexa to crawl my site again, I will have missed all of those months in the archive.

Does anyone have any thoughts as to what I should do regarding allowing Alexa to crawl my site? Any positive or negative feelings on Alexa? On being included in the Internet Archive?

Looking forward to your suggestions! :clapper:

[Andilinks]

I too have mixed feelings about Alexa, but I think there is more to be gained by allowing them. You can put an info file in your www folder specifically for Alexa's use to change or confirm your data.

I think the best use of the Wayback Machine is its ability to prove you had your content at a certain time in case someone tries to accuse you of infringement, or the other side of that coin someone infringes you.

In the last thirty days I've gotten 33 referrals from Alexa, that's not much but it's better than zero.

Andi

[MTDesigns]

You can put an info file in your www folder specifically for Alexa's use to change or confirm your data.

Thanks Andi. Can you be more specific about this info file :)?

[Andilinks]

It has been some time but they instructed me to upload a file called info.txt with this format:

# Contact info submission

url: andilinks.com/
site_owner: Andrea Silver
address1:
address2:
city: Chicago
state: IL
country: Unlisted
postal_code: 60634
phone_number: Unlisted
display_email: x@x.x

The file remains there to this day and this is what Alexa posts and not the whois info which is what they posted originally.

This is probably what they will tell you if you submit your URL here:

http://www.alexa.com/data/details/editor?type=contact

Andi

edited out email addy

[MTDesigns]

Thanks so much Andi :). Do you know if this info.txt file works with anyone else other than Alexa?

[Andilinks]

Do you know if this info.txt file works with anyone else other than Alexa?Not that I'm aware of. Even with Alexa you must notify them that the file exists. They do visit frequently but I do not know if they check this file routinely or only when they are notified. I haven't changed my info since the first time so I have no way of knowing.

Andi

[MTDesigns]

I just realized that Alexa probably got my personal address and phone number from my Amazon.com account. I have no idea why they took the information from there, instead of from the information that I provided to them...

[hobbes]

They probably figure you keep your Amazon account more up to date...

-- First there was Brewster Kahle, then there was WAIS, then Alexa was born, so Brewster sold WAIS and started Alexa Inc; Amazon was just too much of a temptation --