top 100,000 product searches

[krisleech]

hi ya, can any one help me source the top 100,000 product searches on google or froogle or other search engine?

I have come across http://www.google.com/press/zeitgeist.html which is very intresting but only lists the top 10 searches.

And also WordTracker which list the top 500 searches.

Im looking for the stuff lower down really.

Thanks, K.

[georgeek]

K -

Zeitgeist and WT are about as close as you are going to get because that sort of data is kept secret by search engines for commercial reasons.

Occasionally you can get some relevant information from a friendly Google rep just before you commit to $100k/month on AdWords but nothing like the sort of information you want.

- George

[Andilinks]

I too am doing this kind of research and like most who do this I don't want to share my results because that would only dilute the search pool and devalue my work.

But I will point you here because it seems to be a bit further along than you are right now, based on what you've told us:

http://directory.google.com/Top/Computers/Internet/Web_Design_and_Development/Authoring/Online_Tools/Keywords/

I hope this helps. I will be posting some results on my site in the next few weeks but I have nothing to share at the moment except to say that by posting test pages which are then indexed by the search engines and begin attracting searchers you can peer into the "black box."

Andi

[georgeek]

Andi -

krisleech asked for the top 100,000 product searches which is not the same as keyword/phrase research.

Theoretically what krisleech wants is obtainable simply by listing the top 100,000 searches in Froogle (http://froogle.google.com/) and no doubt Google has such a list but they are not going to part with it!

Establishing what keywords/phrases are used when searching for say a 'digital camera' is a whole different ball game.

- George

[Andilinks]

Establishing what keywords/phrases are used when searching for say a 'digital camera' is a whole different ball game. I don't want to belabor the 'ball game' metaphor, the two are not that far apart. True, there is a distiction but one exists within the other not as a separate entity. Isolating Froogle results would further narrow the research but you'd have to hack into Google for that which I wouldn't recommend even if I thought it could be done. Even that result would be imperfect.

Approaching the problem from keyword analysis will eventually produce the result Kris is looking for. I won't hold your hand through the whole process, it's something I'm working on too and I am just handing out hints, not giving the whole store away.

But maybe George, telling him it can't be done will just knock him out of the game sooner. That would be a much simpler solution for you and me.

Andi

[georgeek]

Approaching the problem from keyword analysis will eventually produce the result Kris is looking for.Kris said he was looking for "the top 100,000 product searches" there is no way that keyword analysis could provide that information.

As I said before you might get the top 50 Froogal searches (i.e. products) from a friendly Google rep but that would just be a snapshot over a couple of hours. For predictive purpose it wouldn't be much use though, as the rep would probably say as he was handing them over with grin :)

- George

[Andilinks]

Kris said he was looking for "the top 100,000 product searches" there is no way that keyword analysis could provide that information.

Well, 100,000 would require more time and processing power than I'd be willing to expend but a smaller number is what I'm doing nicely. "No way" is your opinion, not mine. But I am pleased that you think I'm doing the impossible, this bodes well for my competitive edge.

Andi

[Andilinks]

Perhaps you're just getting stuck on more traditional notions of "keyword analysis." I am doing more than this, as I hinted in my first post. But enough of this for now, it hadn't occurred to me that others had not yet figured this out. I'll be quiet.

Andi

[georgeek]

Enjoy the dream Andi :wink:

- Michael

[krisleech]

I actually got two replies from google, below is one, the other is basically the same.

What im trying to do is find the top 10,000 better still 100,000 seraches (keyword/phrases). I really want products ie. anything sellable over the net. There is no real difference in general between normal searches and product searches. The search is either for a 'product' or not.

Im not really looking at tradional SEO. I already know how to optmise a page for a keyword/phrase and gain pagerank. Im kinda looking at it from the other end of the stick. ie. What do people search for a LOT but has little competition in the way of SEO. Like how you would use wordtracker to find related keywords/phrases but in reverse.

WordTracker actually sell top 500,000 searches, but i am yet to find out how this data is gathered. http://www.wordtracker.com/topwords/index.html?sid=
Quite extensive also.

Thanks for the link Andi, i have visited some of these already but some good stuff. Yet to fully explore.

Im even wondering if there is something i/we can do with the google API - ie. create some software that does automated seaches on google. I have a programming background.

I also know it is quite possible to get the info me and Andi require. It a case of how. People already have this data. So we can buy it or re-create it.

Cheers Kris.

Hello Kris -

Thank you for your interest in the Google Zeitgeist. We are unfortunately
not able to provide any statistics or data beyond what is published on our
Google Zeitgeist, http://www.google.com/zeitgeist.

Thank you again for your inquiry.

Kind regards,

Daniel Lemin
Google Inc.
+1-650-623-4405
dlemin@google.com

[georgeek]

WordTracker actually sell top 500,000 searches, but i am yet to find out how this data is gathered.It comes from monitoring the searches in metacrawler.com and dogpile.com.

- George

[krisleech]

thanks George, im not familiar with dogpile, does it search multiple search engines and compile the results?
MetaCrawler, do you think this is acurate considering many people/search engines now ignore meta information due to easy spamability?

How acurate do you think wordtrackers results are in general compaired to google? ie. how well does dogpile/metacrawler match google in terms of number of searches for a particular word/phrase?

Cheers Kris.

[Andilinks]

The term "MetaCrawler" and the generic "meta-crawler" which includes Dogpile and other such searches like ixquick and mama.com refers to the fact that they combine the results of many search engines but do not themselves actually posess an index.

"Meta" in this instance has nothing to do with meta-information provided by websites but is the meta of being a combiner of search results.

[georgeek]

How acurate do you think wordtrackers results are in general compaired to google? ie. how well does dogpile/metacrawler match google in terms of number of searches for a particular word/phrase?No one knows because the numbers in WordTracker are not the actual numbers of searches on dogpile/metacrawler but a prediction of what they might have been. Which is why paying close attention to WordTracker numbers is a waste of time. That doesn't stop people using them or WordTracker making money though :smile:

- George

[krisleech]

"Meta" - Got ya.

DogPile searches Overture, Espotting, Mirago, Turbo10, Open Directory and Ask Jeeves as Dogpile's search partners.

Which actually make its result quite relevant. ie. they should be roughly the same as what is searched for on google, i guess it depends on how popular dogpile is.

George - how do you know Wordtracker 'predicts' results and doesnt use actual results from DogPile?

KRis.

[georgeek]

George - how do you know Wordtracker 'predicts' results and doesnt use actual results from DogPile?The 'count' uses the raw data but that doesn't tell you anything except maybe what the small minority of people that actually use these specified less popular engines are doing. The 'predict' figure they show is exactly that. As they say on their website "The 'predict' figure is exactly that - a prediction".

they should be roughly the same as what is searched for on googleI would have thought the opposite. If you divided the universe of searchers into those that use Google, MSN or Yahoo and those that use some other search engine I would have said that there would be a big difference. In addition the Wordtracker data set is obtained from a subset of this 10% which will make it even more skewed imo.

- George

[Andilinks]

If you divided the universe of searchers into those that use Google, MSN or Yahoo and those that use some other search engine I would have said that there would be a big difference. I'm curious as to why you would say that. I realize they represent a very small sliver of the overall demographic, but what makes them so different? Search results on these meta engines are actually better in my experience, though I use Google more often as well.

Andi

[Rich]

In addition the Wordtracker data set is obtained from a subset of this 10% which will make it even more skewed imo.
As long as the sample is sufficently random and represents a statistically significant sample, it does not have to be any certain percentage or from any particular source to allow accurate predictions to be made.

[georgeek]

As long as the sample is sufficently random and represents a statistically significant sample, it does not have to be any certain percentage or from any particular source to allow accurate predictions to be made.You cannot make accurate predications about the behavior of a completely different data set, which is what they do. You can sample a portion of the population of New Zealand for smokers and then estimate the total number of smokers in New Zealand. But you would be foolish if you tried to use the same data set to estimate the number of smokers in the USA or the whole world.

- George

[Andilinks]

You can sample a portion of the population of New Zealand for smokers and then estimate the total number of smokers in New Zealand. But you would be foolish if you tried to use the same data set to estimate the number of smokers in the USA or the whole world. This is quite true, however these kinds of "seat of the pants" extrapolations can be more valuable than mere guesswork when they are massaged with educated estimates based on other reliable information. They can be a worthwhile point of departure even though not exact or totally reliable. The perfect is enemy of the good--positive feedback can at times validate flawed data.

Andi

[georgeek]

This is quite true, however these kinds of "seat of the pants" extrapolations can be more valuable than mere guesswork when they are massaged with educated estimates based on other reliable information.They are certainly more valuable to WordTracker who use "seat of the pants extrapolations" to make money. The numbers they provide are often used by amateurs without any statistical knowledge to arrive at erroneous conclusions.

Lack of elementary knowledge in statistics leads to more misunderstanding in the world than all the religions put together. Have a read of "More Damned Lies and Statistics by Joel Best" or "Beyond Numeracy by John Paulos" and you will see what I mean.

- George

[Andilinks]

Caveat emptor. Erroneous conclusions, sometimes called mistakes are very common, yes.

So you are saying that WordTracker is totally worthless or just flawed? I don't know, I haven't used it, but you seem to be saying that they are making fraudulant claims.

Andi

[georgeek]

you seem to be saying that they are making fraudulant claims.Not at all in fact they are very good at explaining the origins of their data and the methodology behind their 'estimates'. The trouble is that most people don't read it or understand the consequential limitations.

- Michael

[Andilinks]

The trouble is that most people don't read it or understand the consequential limitations. Caveat emptor applies then. Thanks for sounding the warnings though. I prefer to generate my own stats by analyzing the search terms used to arrive at a (semi) controlled content. Over time the sample grows and trends emerge.

In the end though there always must be some element of guesswork, it's just the percentage of error that diminishes with each iteration--it's never zero.

Andi

[krisleech]

Estimates are still useful to my cause. It depends like you guys say how accurate they are and if they reflect the majority.

On the point of if the less popular search engines reflect what is being searched for on the major engines i would have thought yes because on a global / regional scale we must all be searching for towards an average. It would suprise me (which it may) if people who choose to search with less popular search engines search for different stuff than everyone else. But that may be true. Why do they not use google? because they dont know about it? because they dont read computer mags? because they rarely the internet and so still use the default search engine (MSN)?

I guess for now i keep digging.

Are you keeping your experiments under wraps Andi?

Cheers Kris.

[Andilinks]

In my opinion the searches conducted on the "not-Google" engines do not differ significantly from those done with Google. The differences would relate to the factors you mention and would probably skew toward the newbie user. But that is just another "seat-of-the-pants" opinion and not based on any real data. I don't think that the quality of Google's search results warrant their overwhelming dominance in the market. But they have the momentum and are not standing still so they will likely remain dominant for some years to come unless some newcomer can show a significanly improved result and combine it with marketing as clever as Google's. That's a long shot for now.

Analyses that I do use data from searches made on my website and are mostly Google results. I haven't tried to separate the results by engine though I could. But there are mountains of data collecting daily and only a limited number of things I can do with it given the available time and my skills.

I am not now using the data competitively but since that is the eventual goal, yes my results are proprietary. Clues as to where I am going with this do reside on my website but I'd be shocked if anyone is actually interested enough to look at it this way or would be interested unless I began to profit from it and became a competitor in some arena. Staying low-key in that respect for now suits me fine, I will just quietly build the Andilinks brand one visitor at a time. I have lots of time to play with it. :)

Andi

[krisleech]

Exactly i would have thought non-google engines had the same searches, they of course point people to different websites, but the same searches performed. That is my hunch anyway. No actual data to back that up. Because it does'nt matter if you are a newbie / non-conformist you still look for the same stuff as everyone else dont you. Of course there will be the minority doing there own thing but it should be aprox the same. Google and non-google i would thought get the same percentage of searches for say 'britney spears' or 'manchester united'.
However i cannot get the kind of data i need from the non-googlers either, but i have yet to *really* dig.

What kind of page views are you getting for your homepage?

Looks like you have put some serious hard work in your site, i know its taken years for me to get a PR of 5. The next leap up is even harder! I dont think you can do PR 6 thought link swaps, you need people to actually link to you in a non-reciprical manner through blogs, homepages etc. like people would link to ebay, freinds reunited and the like ie. some good solid content or a unique feature. What are your thoughts?

[Andilinks]

What kind of page views are you getting for your homepage? I have recently put more work into the homepage and am seeing a definite increase in traffic. That page is getting 2-300 views/day but unlike the other pages that get that kind of traffic it is mostly unreferred, only 20-30/day referred. So of those referrals a large percentage are now coming back as regulars. I expect that over time this page will accumulate a nice following. I'm also putting more energy into keeping it current, with new features, etc.

I do no link swaps, I get many offers of that but reject them all. A link request has a better chance if they don't offer a swap.

Oddly enough it went to PR5 rather easily way before I expected it a few years ago but getting to six seems near impossible. I do have one PR6 page which has just one incoming link from a PR7 page.

Andi

[krisleech]

That is a great amount of traffic, do you get many people clicking on your google adwords? I have several information/content sites which feature adwords but i dont really get many clicks.

So are most links to your homepage and deeper pages one way? ie. inbound from other sites.

Do you have insite to the people who link to you, have people linked to your site because it is a useful resource as i assum you dont do link campagin or ask for inbound links.

I would have thought it hard for a website like yours to get such a high PR since it has a huge amount of outbound links. I guess it shows that outbound links dont really effect the PR of a site yet many sites have outbound links but block them (i guess using robots.txt or robots META tags) for example shareware directories have no PR (not sure if its zero or not indexed) excpet for the primary pages like the homepage, about us etc. I have SEO'd for shareware sites before and found submitting to shareware sites no advantage for PR.

Sorry about all the questions i hope you dont mind, i find it all intresting. I have a small web design company and we also do SEO. I have good results despite most of our clients being small - medium business with only a few grand to invest. The key i have found is to take your time, dont rush the process even though you have a forumla. Reasearch the indusrty and keywords/phrases well and go from there with long term SEO-ing of pages for choosen phrases as well as submitting to the many directories. But do the research first find the niche! Usually for small businesses this includes a combo of product and regional information.

Cheers Kris.

[Andilinks]

That is a great amount of traffic,I think the potential is much greater, I'm not happy with the current results but am happy with the rate of increase.do you get many people clicking on your google adwords?The site is profitable, but Google's TOS won't let me detail that. I plan to be set up for some kind of ecommerce within the year.So are most links to your homepage and deeper pages one way? ie. inbound from other sites.If they are reciprocal it is by coincidence since I do link to more than 40,000 pages.have people linked to your site because it is a useful resourceI would guess so.hard for a website like yours to get such a high PR since it has a huge amount of outbound links.Some think there is a special category for directories, but Google doesn't treat directories that well, I don't thinkdo SEOI follow SEO in order to passively avoid things that may be penalized but I do not take any active measures whatsoever.But do the research first find the niche!I agree with your philosophy, Andilinks is itself a niche finding research project.

Andi