Monday, September 20, 2004

Web Crawler Alert

If you are experiencing slow blog posting or viewing do not adjust your computer. We have had reports that someone has activated a deep web crawling bot, an anonymous source told us that most of the hits bore:
    User-Agent: blog-preservation-bot v0.1
So it would seem some "do-gooder" out there is making an paranoid attempt at crawling and caching blogs before they're all gone. Although the anonymous tip said that the bot crawled some of his pages over 3000 times, so we suspect it hasn't been perfected yet.

Don't bother updating your robots.txt file, this bot doesn't read it... you can't escape this creepy crawly sorry. As of yet we are unsure of the source of the bot, or who is responsible for these aggressive tactics. No doubt some white hat will find a way to disable this spider for the greater good of the net community.

In my honest opinion, this is a wake up call to those who would think of deleting their own blogs. All your posts could be preserved for all time if you have managed to stay out of search engines so far (more than likely).

Don't go far, no doubt news will breaking again any minute.

0 Comments:

Post a Comment

<< Home