Spider list contains the list of spiders that are recognized by both the Robots.txt File Editor program. You can easily sort spiders by name, country or primary language just by clicking on the appropriate column header. Registered users can also add new spiders or edit the field of the information about existing crawlers.(You just have to contact us at
support@net-promoter.com) Detailed fields of the spider information dialog window also informs about the site domains that are supported by the selected search engine and the spider information URL. The status bar displays the number of selected spiders against the total number of spiders. The spider menu contains the Select By... function which allows you to quickly group spiders by certain criteria (like country and primary language).
You can disallow the access to the resources for the spiders selected on the Spider List Tab. Spiders can be grouped by country, primary language or you can select arbitrary set of spiders. The banned resources are represented as the tree of web site files either on FTP server or saved locally. Unlike other programs Robots.txt Editor supports creating disallow rules by file extensions.
Finally, you can generate robots.txt file by clicking Generate File button.
Log Analyzer Tab
The Log Analyzer detects user-agents for more than 150 spiders! It is also a good idea to detect spiders automatically and add them to your spiders list. The visits of crawlers are represented as a tree ordered by date, spider or the requested page. You can limit the output to the selected period.
Apart from giving the detailed information on the visitors, the tree will highlight:
- spiders that did not request robots.txt file
- images
- redirections and not-found pages (404 errors)
- violations of robots.txt file restricti
You can export Analyzer results into Microsoft Excel CSV table, XML file or Apache standard W3C log file
