HolyCowDude/0.1 Feeds Searcher

The spider was created to collect the RSS feeds links from all the websites around the globe, it is designed to prepare these feeds for crawling by the brother spider called "HolyCowDude Feeds Indexer" which is intended to index the content of the feeds.

The bot is getting the websites links from the Pingoat's pinged websites list and the iPings's pinged websites list. That's the main source of finding people who want to get their feeds spidered and indexed.

Also, the spider was trained to look into the robots.txt file of each spidered website and avoid going into the prohibited files or folders of the target website.

First screenshot of the HolyCowDude:

This page was last time edited on: May 20, 2006: 4:15

Back to the main page