Free browser-based tool
Robots.txt Generator
Draft a simple robots.txt file for WordPress and small websites with user-agent rules, allow paths, disallow paths, sitemap URL, crawl delay, and common admin/search rules.
Enter your details
Result
Your result will appear here.
Use this result
Draft crawl directives for named user agents, public and blocked paths, parameter patterns, and sitemap discovery. The result should control crawling without accidentally hiding pages that need indexing.
Worked example
For WordPress, allow public content, disallow /wp-admin/ while allowing /wp-admin/admin-ajax.php, add the canonical sitemap URL, and review whether internal search or parameter URLs need crawl controls.
Check before using the result
Do not use robots.txt as a reliable way to remove an indexed page. Avoid blocking CSS or JavaScript needed for rendering, and check that wildcard syntax matches the crawler you intend to address.
What this tool does not establish
Use page-level noindex, authentication, redirects, or removal tools for indexing and access problems. Test important URLs in Search Console after publishing.
Questions about this task
Does Disallow remove a page from Google?
Not reliably. A blocked URL can still appear without a snippet if Google finds links to it.
Can robots.txt contain more than one sitemap?
Yes, but each sitemap URL should be complete and should list canonical indexable URLs.
Related tools for the next step
Continue with your files or text
Check a revised title, description or content draft with a separate word and character counter.
Count words and characters in your draft ↗