ToolsHub4u

Write a robots.txt that says exactly what you mean

Write the allow and disallow paths, add a sitemap and optional AI crawler groups, and the generator checks the paths and shows the finished robots.txt.

  • Syntax checks
  • AI crawler options
  • Download robots.txt
Ready SEO & Web
Robots.txt is a request, not a security controlWell-behaved crawlers follow it; others ignore it. It does not hide a page from people, and a disallowed URL can still be indexed if other sites link to it. Use noindex for pages that must stay out of results.
1Enter
2Adjust
3Copy result

Robots.txt Generator

1

Rules and options

One path per line. Each must start with a slash. Wildcards (*) and an end anchor ($) are supported by Google and Bing.

Allow rules override a broader disallow when they are more specific. Use them sparingly.

Disallow everything is a staging-site setting. Never leave it on a live site.

Block AI crawlers

Blocks the named crawler from crawling the whole site. It does not remove content already used, and it does not block AI answers that cite pages without crawling them.

No data yet Live
2

robots.txt

Rules written and checked

Disallow rules

5

robots.txt

  • 5 disallow pathsEach rule blocks crawling of matching URLs. It does not remove them from the index.
  • Sitemap is an absolute URLA sitemap line helps crawlers find the file, though Search Console submission is more reliable.
  • Site URL is validThe site URL is only used in the comment line; robots.txt rules are always relative to the host that serves the file.
  • Everything is allowed by defaultOnly the listed paths are blocked.
  • Every path is validPaths start with a slash and contain no spaces.
Rules in this file
GroupDirectivePathEffect
All crawlersDisallow/wp-admin/Not crawled
All crawlersDisallow/cart/Not crawled
All crawlersDisallow/checkout/Not crawled
All crawlersDisallow/search?Not crawled
All crawlersDisallow/*?s=Not crawled
All crawlersAllow/wp-admin/admin-ajax.phpCrawlable, overrides a broader block
Disallow rules
5
Allow rules
1
AI bots blocked
0
Sitemap
Set

Save this as robots.txt at the root of the domain. Test it with the robots.txt report in Search Console or Bing Webmaster Tools before relying on it.

A robots.txt file is public and can be read by anyone. Do not put private paths in it that you would not want discovered, and do not use it to protect a page.

This page writes a robots.txt file from a short set of rules. You give it the site URL, the paths to block, any paths to allow back, an optional sitemap and optional groups for AI crawlers. It normalises the paths, rejects the ones it cannot use and shows the finished file.

robots.txt is a small file with a large blast radius. A stray Disallow: / can stop a live site being crawled, and a rule written for the wrong path can quietly block a section you meant to keep. The page checks what it can, but the file still needs a human review before it replaces the one on the server.

What robots.txt can and cannot do

A robots.txt file tells crawlers which URLs they may fetch. It does not remove a page from Google's index. Google can index a disallowed URL, usually without a snippet, if it finds links to it elsewhere. To keep a page out of search results, use a noindex robots meta tag or password protection, not a disallow.

The file must sit at the root of the host, at https://example.com/robots.txt. A copy in a subdirectory is not read. It applies only to the host, protocol and port that serve it, so it does not cover a subdomain or the http version of an https site. Paths are case-sensitive, and Google enforces a 500 KiB size limit.

Google supports four fields: user-agent, allow, disallow and sitemap. Other fields are ignored, including crawl-delay. The file is public: anyone can read it, so do not list private paths you would not want discovered.

How the generator builds the file

With the default set to allow, the file starts with User-agent: *, writes one Disallow line per blocked path, then one Allow line per exception. If there are no disallow paths it writes an empty Disallow: line, which is valid and means nothing is blocked. The optional crawl delay follows the group, then any AI crawler groups, then the sitemap line.

With the default set to disallow, it writes Disallow: / and then the allow lines only. That is the correct shape for a staging site: block everything, then open the few paths that must remain reachable, such as an admin AJAX endpoint.

Path normalisation adds a leading slash if one is missing. A line containing a space is rejected. A line beginning with // is rejected as a full URL rather than a path. The AI options write a separate User-agent group with Disallow: / for GPTBot, ClaudeBot, CCBot, Google-Extended or PerplexityBot.

A worked example

Five paths were blocked, one was allowed back and a sitemap was added. The rules in the file were:

The generated file for the allow-all example
LineEffect
User-agent: *Applies to every crawler
Disallow: /wp-admin/Blocks the admin directory
Disallow: /cart/Blocks the cart
Disallow: /checkout/Blocks checkout
Disallow: /search?Blocks search URLs
Disallow: /*?s=Blocks the query parameter pattern
Allow: /wp-admin/admin-ajax.phpRe-opens one file inside a blocked directory
Sitemap: https://toolshub4u.com/sitemap.xmlPoints to the sitemap

The stats read five disallow rules, one allow rule, no AI bots and the sitemap set. The table above the output labelled each row Not crawled or Crawlable, overrides a broader block. That is the intended behaviour: a longer, more specific allow can override a broader disallow.

A second run switched the default to disallow and kept /cart/ in the disallow box. The file contained Disallow: / and Allow: /wp-admin/admin-ajax.php, with no /cart/ line, because everything is already blocked. The table, however, showed Exception / Allow / /cart/ / Crawlable, and the stats still counted one disallow rule. Both are wrong for that mode: the path is not in the file and is not crawlable.

Check the file, not the tableWhen the default is disallow, the disallow list is not written. Read the output box before uploading. A table row saying a path is an allowed exception does not put it in the file.

The full-URL trap

A path must start with a slash. If you paste https://example.com/private, the generator prepends a slash and accepts the result as /https://example.com/private. That is a valid-looking path that will never match the URL you meant. It is the most expensive mistake this form allows, because the file passes every check and blocks nothing useful.

Two other lines were rejected in the same run: //cdn.example.com/x because it looks like a full URL, and /has space because a path cannot contain a space. Both were listed with a reason and left out of the file. Google ignores rules without a valid path, so a rejected line is safer than a mangled one.

Always paste the path portion only. If you are copying from a browser address bar, delete the scheme and host before pasting, and check that the first character is a single slash.

Crawl-delay and AI crawler groups

The crawl-delay option writes Crawl-delay: 1. Google does not support the field and ignores it. Bing and Yandex have historically treated it differently, so it is not a portable way to slow a crawler. Use Search Console's crawl rate setting for Google, and treat the line as a request to other engines, not a guarantee.

Blocking an AI crawler with Disallow: / stops that bot fetching pages. It does not remove content already collected, and it does not stop an AI feature citing a page it found through another route. The checklist says this, and it is worth reading before treating the option as an opt-out from all AI use.

A sitemap value must be a full URL. A relative value such as sitemap.xml is left out of the file and written as a comment, and the stats report the sitemap as missing. That is the right call: Google does not resolve a relative sitemap line.

Publishing and testing

Save the output as robots.txt at the site root, overwriting the old file only after a review. Google may cache a robots.txt for up to 24 hours, so a correction is not instant. Test the live file in the robots.txt report in Search Console or Bing Webmaster Tools, and fetch it in a private window to confirm it is publicly readable.

Keep the file short. Most sites need a handful of disallow lines at most, and every extra rule is another chance to block something by accident. If the goal is to manage which pages are indexed, the sitemap generator and a noindex tag are better tools for the job.

Once the file is live, the log file analyser shows whether crawlers are still requesting the paths you blocked. A path that keeps appearing after a block may be cached, linked from elsewhere, or simply not covered by the rule you wrote.

Questions

Common questions

Short answers about this tool and its limits.

Will blocking a page remove it from Google?

No. A disallow stops crawling, not indexing. Google can still index the URL, usually without a snippet, if it finds links to it. Use a noindex meta tag or password protection to keep a page out of results.

Why was my full URL turned into a path?

The generator adds a leading slash to anything that does not start with one. Pasting https://example.com/private produces /https://example.com/private, which is accepted but will not match the page. Paste only the path, starting with a single slash.

Does Google support Crawl-delay?

No. Google's robots.txt documentation lists only user-agent, allow, disallow and sitemap, and says other fields such as crawl-delay are not supported. Use the crawl rate setting in Search Console for Google.

Why is a path missing when the default is disallow?

Because that mode writes Disallow: / and only the allow lines. The disallow list is not written, since everything is already blocked. The table may still label those paths as exceptions, which is a display bug; check the output file.

Does a relative sitemap URL work?

No. Google requires a fully qualified URL, including the scheme and host. The generator leaves a relative value out and writes a comment explaining why, and the stats report the sitemap as missing.

Do the AI bot groups stop AI using my content?

They stop those crawlers fetching pages. They do not remove content already collected, and they do not stop an AI feature citing a page found another way. Treat the groups as a crawling choice, not a complete opt-out.

Can I use wildcards in a path?

The generator writes what you type, so /*?s= and /*.php$ are passed through. Google supports * and $ in allow and disallow rules; other crawlers may not. Test the pattern before relying on it.

Where does the file go?

At the root of the host, named robots.txt. It cannot live in a subdirectory, and each host or subdomain needs its own file. Google may cache it for up to 24 hours after a change.