Robots.txt Generator
What this tool does
Select a user-agent (or use * for all bots), add Allow and Disallow rules for specific paths, and optionally set a crawl-delay and sitemap URL. The live preview on the right shows the exact robots.txt content. Use a preset to start quickly, then adjust rules for your site's structure.
How to use
- Click a preset (e.g. WordPress) or start with Allow All.
- Select a User-Agent to target — use * for all crawlers.
- Add Disallow rules for paths you want to block.
- Optionally add your Sitemap URL in the Options section.
- Click Copy or Download to get the file.
Frequently Asked Questions
What is robots.txt?
robots.txt is a plain text file at the root of your website that tells crawlers which pages they may or may not access. It follows the Robots Exclusion Protocol and is the first file crawlers check. It controls crawl budget but is not a security mechanism — pages can still be accessed directly.
What does Disallow: / mean?
Disallow: / blocks the crawler from accessing any page on your site. It is the most restrictive rule. Use it only if you want to completely prevent a bot from crawling your website — for example, blocking AI training crawlers like GPTBot.
What is the difference between Allow and Disallow?
Disallow prevents a crawler from accessing a path. Allow explicitly permits access, and can be used to create exceptions inside a disallowed path. For example: Disallow /admin/ plus Allow /admin/public/ lets crawlers access only the public subdirectory.
What is Crawl-delay?
Crawl-delay tells bots how many seconds to wait between requests. It reduces server load during crawling. Note: Google ignores the Crawl-delay directive — use Google Search Console to manage Googlebot's crawl rate instead.
Where should I put the robots.txt file?
The robots.txt file must be placed at the root of your domain — for example https://example.com/robots.txt. It cannot be in a subdirectory. Each subdomain requires its own robots.txt.