garyamort
08-21-2007, 08:17 AM
In the vein of informing the customer, since FQ is in the position of dealing with malicious/badly designed searchbots - it would be nice if they could post a list of the search engine identifications they have discovered, linked back to whom the engine is searching for where appropriate.
That way site owners that want to restrict search engines would have a list that we could use and periodically update from as well as scouring our own logfiles and the internet.
Since generally the point of detecting a search engine is to deny it access to the site, slow it down, or give it specialised access, it would be a win/win for everyone. As the more site owners limit search engines, the lower the impact on the community servers of those search engines.
That way site owners that want to restrict search engines would have a list that we could use and periodically update from as well as scouring our own logfiles and the internet.
Since generally the point of detecting a search engine is to deny it access to the site, slow it down, or give it specialised access, it would be a win/win for everyone. As the more site owners limit search engines, the lower the impact on the community servers of those search engines.