Free browser-based tool
Robots.txt Generator
Draft a simple robots.txt file for WordPress and small websites with user-agent rules, allow paths, disallow paths, sitemap URL, crawl delay, and common admin/search rules.
Enter your details
Result
Your result will appear here.
How this helps
Draft a simple robots.txt file for WordPress and small websites with user-agent rules, allow paths, disallow paths, sitemap URL, crawl delay, and common admin/search rules. The goal is to make the first working draft clear enough to review, share, and improve instead of starting from a blank document.
Real-world example
A basic WordPress robots.txt often disallows admin pages, allows admin-ajax, and points crawlers to the XML sitemap.
Common mistakes
- Blocking important pages by accident.
- Forgetting the sitemap line.
- Using robots.txt to hide sensitive information.
Robots.txt Generator: what the result means
Draft crawl directives for named user agents, public and blocked paths, parameter patterns, and sitemap discovery. The result should control crawling without accidentally hiding pages that need indexing.
A realistic way to use it
For WordPress, allow public content, disallow /wp-admin/ while allowing /wp-admin/admin-ajax.php, add the canonical sitemap URL, and review whether internal search or parameter URLs need crawl controls.
Checks that prevent a bad result
Do not use robots.txt as a reliable way to remove an indexed page. Avoid blocking CSS or JavaScript needed for rendering, and check that wildcard syntax matches the crawler you intend to address.
Choose the correct control for the SEO outcome
Robots.txt manages crawling, not every indexing problem. Match the rule to the outcome before publishing a sitewide block.
| Goal | Use this control | Important caution |
|---|---|---|
| Reduce crawling | A narrow Disallow rule for a repeatable path pattern | Test the pattern so valuable pages and assets remain crawlable. |
| Remove a page from search | A crawlable noindex directive, removal workflow, or deletion as appropriate | A blocked URL may prevent a crawler from seeing noindex. |
| Protect private content | Authentication and access control | Robots.txt is public and does not secure confidential URLs. |
| Declare the sitemap | An absolute Sitemap line using the canonical sitemap URL | Confirm the sitemap returns 200 and contains canonical indexable URLs. |
Before you act: test several affected and unaffected URLs, keep the rule set as small as possible, and monitor crawl or indexing changes in Search Console.
When this tool is not enough
Use page-level noindex, authentication, redirects, or removal tools for indexing and access problems. Test important URLs in Search Console after publishing.
Frequently asked questions
Does Disallow remove a page from Google?
Not reliably. A blocked URL can still appear without a snippet if Google finds links to it.
Can robots.txt contain more than one sitemap?
Yes, but each sitemap URL should be complete and should list canonical indexable URLs.