This page writes a robots.txt file from a short set of rules. You give it the site URL, the paths to block, any paths to allow back, an optional sitemap and optional groups for AI crawlers. It normalises the paths, rejects the ones it cannot use and shows the finished file.
robots.txt is a small file with a large blast radius. A stray Disallow: / can stop a live site being crawled, and a rule written for the wrong path can quietly block a section you meant to keep. The page checks what it can, but the file still needs a human review before it replaces the one on the server.
What robots.txt can and cannot do
A robots.txt file tells crawlers which URLs they may fetch. It does not remove a page from Google's index. Google can index a disallowed URL, usually without a snippet, if it finds links to it elsewhere. To keep a page out of search results, use a noindex robots meta tag or password protection, not a disallow.
The file must sit at the root of the host, at https://example.com/robots.txt. A copy in a subdirectory is not read. It applies only to the host, protocol and port that serve it, so it does not cover a subdomain or the http version of an https site. Paths are case-sensitive, and Google enforces a 500 KiB size limit.
Google supports four fields: user-agent, allow, disallow and sitemap. Other fields are ignored, including crawl-delay. The file is public: anyone can read it, so do not list private paths you would not want discovered.
How the generator builds the file
With the default set to allow, the file starts with User-agent: *, writes one Disallow line per blocked path, then one Allow line per exception. If there are no disallow paths it writes an empty Disallow: line, which is valid and means nothing is blocked. The optional crawl delay follows the group, then any AI crawler groups, then the sitemap line.
With the default set to disallow, it writes Disallow: / and then the allow lines only. That is the correct shape for a staging site: block everything, then open the few paths that must remain reachable, such as an admin AJAX endpoint.
Path normalisation adds a leading slash if one is missing. A line containing a space is rejected. A line beginning with // is rejected as a full URL rather than a path. The AI options write a separate User-agent group with Disallow: / for GPTBot, ClaudeBot, CCBot, Google-Extended or PerplexityBot.
A worked example
Five paths were blocked, one was allowed back and a sitemap was added. The rules in the file were:
| Line | Effect |
|---|---|
User-agent: * | Applies to every crawler |
Disallow: /wp-admin/ | Blocks the admin directory |
Disallow: /cart/ | Blocks the cart |
Disallow: /checkout/ | Blocks checkout |
Disallow: /search? | Blocks search URLs |
Disallow: /*?s= | Blocks the query parameter pattern |
Allow: /wp-admin/admin-ajax.php | Re-opens one file inside a blocked directory |
Sitemap: https://toolshub4u.com/sitemap.xml | Points to the sitemap |
The stats read five disallow rules, one allow rule, no AI bots and the sitemap set. The table above the output labelled each row Not crawled or Crawlable, overrides a broader block. That is the intended behaviour: a longer, more specific allow can override a broader disallow.
A second run switched the default to disallow and kept /cart/ in the disallow box. The file contained Disallow: / and Allow: /wp-admin/admin-ajax.php, with no /cart/ line, because everything is already blocked. The table, however, showed Exception / Allow / /cart/ / Crawlable, and the stats still counted one disallow rule. Both are wrong for that mode: the path is not in the file and is not crawlable.
The full-URL trap
A path must start with a slash. If you paste https://example.com/private, the generator prepends a slash and accepts the result as /https://example.com/private. That is a valid-looking path that will never match the URL you meant. It is the most expensive mistake this form allows, because the file passes every check and blocks nothing useful.
Two other lines were rejected in the same run: //cdn.example.com/x because it looks like a full URL, and /has space because a path cannot contain a space. Both were listed with a reason and left out of the file. Google ignores rules without a valid path, so a rejected line is safer than a mangled one.
Always paste the path portion only. If you are copying from a browser address bar, delete the scheme and host before pasting, and check that the first character is a single slash.
Crawl-delay and AI crawler groups
The crawl-delay option writes Crawl-delay: 1. Google does not support the field and ignores it. Bing and Yandex have historically treated it differently, so it is not a portable way to slow a crawler. Use Search Console's crawl rate setting for Google, and treat the line as a request to other engines, not a guarantee.
Blocking an AI crawler with Disallow: / stops that bot fetching pages. It does not remove content already collected, and it does not stop an AI feature citing a page it found through another route. The checklist says this, and it is worth reading before treating the option as an opt-out from all AI use.
A sitemap value must be a full URL. A relative value such as sitemap.xml is left out of the file and written as a comment, and the stats report the sitemap as missing. That is the right call: Google does not resolve a relative sitemap line.
Publishing and testing
Save the output as robots.txt at the site root, overwriting the old file only after a review. Google may cache a robots.txt for up to 24 hours, so a correction is not instant. Test the live file in the robots.txt report in Search Console or Bing Webmaster Tools, and fetch it in a private window to confirm it is publicly readable.
Keep the file short. Most sites need a handful of disallow lines at most, and every extra rule is another chance to block something by accident. If the goal is to manage which pages are indexed, the sitemap generator and a noindex tag are better tools for the job.
Once the file is live, the log file analyser shows whether crawlers are still requesting the paths you blocked. A path that keeps appearing after a block may be cached, linked from elsewhere, or simply not covered by the rule you wrote.