Top > Authoring > Online Tools > robots.txt
Popular Tags
Web Links
robots.txt
Analyzes a robots.txt file searching for syntax and "logical" errors, and shows a summary of what effect it will have.
Date Added: Apr 6, 2010 Hits: Rating: 0.00 Votes: 0
Category:
Random Site Quote:
If you care about validation, this robots.txt validator is a tester that will check your robots.txt file searching for syntax errors
Robots.txt files (often erroneously called robot.txt, in singular) are created by webmasters to mark (disallow) files and directories of a web site that search engine spiders (and other types of robots) should not access. This robots.txt checker is a "validator" that analyzes the
Robots.txt files (often erroneously called robot.txt, in singular) are created by webmasters to mark (disallow) files and directories of a web site that search engine spiders (and other types of robots) should not access. This robots.txt checker is a "validator" that analyzes the
Date Added: Apr 6, 2010 Hits: Rating: 0.00 Votes: 0
Member Reviews
Visitor Ratings 
Google PR
Online tool for syntax verification to robots.txt files, provided by Simon Wilkinson.
Date Added: Apr 6, 2010 Hits: Rating: 0.00 Votes: 0
Category:
Random Site Quote:
Robots.txt syntax checker
Writing robots.txt files is relatively straightforward. However, its still
fairly common to get some small part of the syntax wrong, and then to find
that a robot has wandered all over your site because of it.
This robots.txt syntax checker checks the contents of a
waiting for hand_moderation
waiting for hand_moderation
Date Added: Apr 6, 2010 Hits: Rating: 0.00 Votes: 0
Member Reviews
Visitor Ratings Google PR
Creator and validator of robots.txt files.
Date Added: Apr 6, 2010 Hits: Rating: 0.00 Votes: 0
Category:
Random Site Quote:
Your Web Hosting Solution
robots.txt Tutorial - Easy Tools
These tools are provided to help you create a new robots.txt file for your site or
Simply select a robot (or all ) and enter the path or paths you wish to
Create a custom robots.txt file
Paths/Directories to exclude
To completely ban a
waiting for hand_moderation
waiting for hand_moderation
Date Added: Apr 6, 2010 Hits: Rating: 0.00 Votes: 0
Member Reviews
Visitor Ratings Google PR
Information on the use of robots.txt as well as a robots.txt generator and a list of search engine robots.
Date Added: Apr 6, 2010 Hits: Rating: 0.00 Votes: 0
Category:
Random Site Quote:
Information on the robots.txt and how it effects your website. Also includes a free robots.txt generator
First some back ground on search engine robots Web indexing robots are used by many search engines such as Google, Inktomi, AltaVista and others. These web indexing robots are also known as spiders. These spiders/robots are the tools used by engines to harvest data for their search engines. When
First some back ground on search engine robots Web indexing robots are used by many search engines such as Google, Inktomi, AltaVista and others. These web indexing robots are also known as spiders. These spiders/robots are the tools used by engines to harvest data for their search engines. When
Date Added: Apr 6, 2010 Hits: Rating: 0.00 Votes: 0
Member Reviews
Visitor Ratings Google PR
Robots.txt generator designed for public use. Includes tutorial.
Date Added: Apr 6, 2010 Hits: Rating: 0.00 Votes: 0
Category:
Random Site Quote:
robots.txt generator designed by an SEO for public use. Includes tutorial.
Robot Control Code Generation Tool If you know of a robot that should be added to this list and we will verify and add Default - All Robots are: The path is relative to root and must contain a trailing "/" Now, copy and paste this text into a blank text file called
Robot Control Code Generation Tool If you know of a robot that should be added to this list and we will verify and add Default - All Robots are: The path is relative to root and must contain a trailing "/" Now, copy and paste this text into a blank text file called
Date Added: Apr 6, 2010 Hits: Rating: 0.00 Votes: 0
Member Reviews
Visitor Ratings Google PR
Tool that fetches robots.txt from a given URL and parses it against the URL. Includes different user-agents.
Date Added: Apr 6, 2010 Hits: Rating: 0.00 Votes: 0
Category:
Random Site Quote:
Robots.txt Analysis - Check whether your site can be accessed by Robots (Google,Yahoo,MSN)
With this free tool you will be able to test and analyze URL against robots.txt file. The tool will fetch the robots.txt file from the domain and it will parse (pattern matching supported) it to check whether
waiting for hand_moderation
waiting for hand_moderation
Date Added: Apr 6, 2010 Hits: Rating: 0.00 Votes: 0
Member Reviews
Visitor Ratings Google PR
Main Category
143
302
42
2
9188
87
636
2020
15
26
1598
88
72
Pagerank Statistics
Directory Statistics
Links: 14219
Categories: 368
Yahoo! News
Top 10
