<<< Robots.txt Editor Overview
System Requirements >>>

In order to make the program easier to work with, its operations are implemented in the form of project management. Each project corresponds to one web site. You may save the project to continue the work later or transfer the data to another computer.

The program consists of two main modules:

• the Robots.txt standardized file editor for setting the spiders policy, and
• a powerful Spider Log Analyzer

The first module, Robots.txt Editor, is a visual editor for the robots.txt file that contains spidering instructions for search robots. This powerful tool enables you to:

• keep track of the spider list, update it and add new spiders to it by yourself
• record the information on each spider, including its address and all User-Agents (nicknames it uses when visiting your site)
• select spiders by the country of their location and main language of their search systems
• set the policy for each spider and selected groups of spiders whenever necessary
• disallow access to the entire site for a selected spider easily
• allow access to certain areas of your site only for those search systems, to requirements of which these pages have been specially optimized
• disallow access to files of selected format, for example, to service files stored at your server
• import and edit existing robots.txt files

The second module, Spider Log Analyzer, is a set of tools and step-by-step wizards that will allow you to perform a thorough analysis of spider visits to your web site. Using the Spider Log Analyzer module, you'll be able to perform the following operations on log files:

• load them from FTP server or local disk
• store in the project files for each specific website
• export to Microsoft Excel CSV, XML, and .log formats

The Spider Log Analyzer module will also let you to:

• group spider visits by date, path, and spider
• review a graphic presentation of all successful and unsuccessful spider visits to individual pages and files of your site, including robots.txt file
• watch the spider visits trends in different time frames
• generate various reports on the analyzed files

Using these modules will give you not only the possibility to record the spider activity, but also a possibility to control it.

Our program also offers you additional useful features:

• uploading files to your web server via FTP
• a list of Known User-Agents that are not spiders
• capability of working via proxy server or behind a firewall

<<< Robots.txt Editor Overview
System Requirements >>>