Create robots.txt rules that communicate crawler access clearly, protect low-value paths, preserve important pages, and link sitemap discovery without blocking the pages you want indexed.
Run the tool to see analysis, exports, and next actions here.
The generator creates a readable robots.txt draft based on site URL, sitemap URL, crawler identity, allow rules, disallow rules, and optional crawl delay. It favors clarity over clever blocking patterns.
Use it before launching a new site, restructuring private paths, cleaning crawl traps, or reviewing whether faceted, checkout, search, and internal pages should be crawled.
Robots.txt is a crawl directive, not an indexing guarantee. A disallowed URL can still be discovered through links, and an allowed URL can still be noindex or canonicalized elsewhere.
The most damaging robots mistakes are broad rules that block assets, localized sections, product pages, or the entire site. A small syntax change can become a traffic incident.
Review the draft, test sample URLs, and crawl critical paths before deploying. The safest robots file is one that is easy to explain and easy to validate.
Quick answers for crawl planning, metadata QA, and SEO handoffs.
Not reliably. Robots.txt controls crawling, not indexing. If you need a page removed from the index, use noindex on a crawlable page or appropriate removal workflows.
Usually yes for public search crawlers, but make sure broad rules do not also block product, collection, or documentation paths that should rank.
Yes, adding sitemap directives helps crawlers discover sitemap files, especially on larger sites or sites with multiple sitemap indexes.
Only use it when server load requires it. Some crawlers ignore crawl delay, and overly restrictive delays can slow discovery on sites that need frequent updates.