Robots.txt Generator
Generate a valid robots.txt with allow, disallow and sitemap directives.
About the Robots.txt Generator
This Robots.txt Generator builds a valid robots.txt file from simple inputs — the user-agent, the paths you want to disallow or allow, an optional crawl-delay, and your sitemap location. A correct robots.txt keeps crawlers out of pages that should not be indexed without accidentally blocking the ones that should.
Frequently asked questions
It must sit at the root of your domain, reachable at yourdomain.com/robots.txt. Search engines only look for it there, so placing it in a subfolder has no effect.
Not on its own. Disallow stops crawling, but a page can still appear in results if other sites link to it. To keep a page out of the index entirely, use a noindex meta tag instead.