Robots.txt generator & validator.
Write crawler directives, check actual parser diagnostics and export a clear robots.txt file. Understand what the rules can — and cannot — do.
Build or check robots.txt
Create a controlled ruleset or paste existing robots.txt text. Validation uses the Site Discovery Engine parser, not a fabricated local score.
Clear rules, without guesswork.
Enter rules or text, then use the actual SDE generator or validator to inspect syntax and diagnostics.
CRAWL DIRECTIVES, EXPLAINED
Tell compliant crawlers where to go.
Robots.txt is a plain-text file served at a site's root, usually /robots.txt. It uses user-agent groups and Allow/Disallow path rules. Our tool passes the rules to Site Discovery Engine rather than guessing whether a file is valid.
The generator returns the engine's output only when valid. The validator reports actual diagnostics without fabricating a site audit or crawler-specific path test.
Set the user-agent and permitted or disallowed path patterns.
Inspect parser errors and warnings, including risky broad restrictions.
Download robots.txt and deploy it to the root of your own site.
FREQUENT QUESTIONS
What does robots.txt control?
Does robots.txt hide private pages?
No. It is not security, authentication or authorization. A disallowed URL can still be directly visited. Never use it to protect secrets or private content.
Will Disallow remove a page from Google?
Not reliably. A crawl block does not guarantee deindexing. For removal, use the appropriate indexing controls and check that crawlers can access them.
Are all directives supported equally by Googlebot?
No. Support for optional directives and wildcard matching varies by crawler. Validation confirms Site Discovery Engine's parser behavior, not exact Googlebot matching or ranking changes.
Can the validator fetch an existing site's robots.txt?
This version accepts pasted text only. There is no arbitrary URL fetch from this tool; paste the file contents to validate safely.
Where should I upload the generated file?
Publish it at the site's root, such as https://example.com/robots.txt. Robots files have rules tied to their exact scheme, host and port.
COMPLETE TECHNICAL SEO