Robots.txt Generator – Control Search Engine Crawling
Choose your user agents, block or allow paths, set a crawl delay and point to your sitemap — then copy or download a clean, valid robots.txt file.
20+SEO Tools
100%Free
0Signup Required
Robots.txt Generator
Build a valid robots.txt file for your website. Paths are automatically prefixed with a leading slash — one path per line.
one per line
optional — overrides a broader Disallow
Upload this file as robots.txt to your site root — it must be at https://yoursite.com/robots.txt.
Quick Answer
robots.txt is a small text file placed at your website's root that tells search engine crawlers which pages they may or may not crawl. Use the generator above to choose your user agents, block or allow paths, add a crawl delay and point crawlers to your sitemap — then copy or download the finished file.
How robots.txt Works
When a crawler like Googlebot visits your site, it first requests robots.txt. The file contains simple rules — which user agents the rules apply to, and which paths are disallowed or allowed. It is a polite protocol, not a security mechanism: crawlers choose to honor it, and it does nothing to stop humans or malicious bots.
The Key Directives
- User-agent — which crawler the rules apply to.
*means all crawlers. - Disallow — blocks crawling of a path. An empty value allows everything.
- Allow — permits a path, mainly to override a broader Disallow rule.
- Crawl-delay — how many seconds a crawler should wait between requests (not supported by Google, but respected by some others).
- Sitemap — points crawlers to your XML sitemap URL.
What to Block (and What Not To)
Good candidates for blocking:
- Admin and login areas (
/admin,/wp-admin). - Search result pages and internal search (
/search). - Duplicate or thin content like tags, filters, or pagination you don't want crawled.
- Temporary or test directories (
/tmp,/staging).
Don't block CSS, JavaScript, or images that your pages need to render correctly — Google has said it needs these to understand your layout and to detect responsive design.
A Common Mistake: Blocking vs. Removing
Blocking a page with robots.txt stops crawling — but if the page is linked from elsewhere, Google can still index it (showing only its URL). To genuinely remove a page from search results, use the noindex meta tag instead, or remove the page entirely. A common pattern is to block crawling of heavy assets while using noindex for pages you want gone from results.
How to Use the Generator
- Pick a user agent — all crawlers, Googlebot, Bingbot, or a custom bot.
- List paths to block in Disallow — the generator adds the leading slash for you.
- Add Allow rules to override any broad blocks (optional).
- Set a crawl delay if your server can't handle aggressive crawling (optional).
- Add your sitemap URL, then copy or download the file.
- Upload it to your site root and test it with the robots.txt tester in Google Search Console.
Conclusion
robots.txt is a tiny file with an outsized effect on how search engines treat your site. Get it right — block what should stay private, allow everything else, and point crawlers to your sitemap — and your crawl budget goes where it matters. Generate yours above and test it in Search Console today.
Reviews
What Our Users Say
Trusted by thousands of users worldwide
"Our staging site kept getting indexed and leaking into Google results. Ten seconds with this generator — block /staging, add the sitemap, download — and the problem was gone. Exactly the kind of tool I wish I'd found years ago."
"I appreciated that the generator explains what each directive means — most tools just spit out a file. I learned that blocking pages doesn't unindex them, which saved me from a costly mistake on my client's site."
"My WordPress admin was being hammered by bots. I generated a clean robots.txt with the admin blocked and a crawl delay, uploaded it, and my server load dropped noticeably within days. Simple fix, big difference."
FAQ
Frequently Asked Questions
Quick answers to common questions
robots.txt is a plain-text file at your site root that tells search engine crawlers which parts of your site they may or may not crawl.
It must live at the root of your domain, e.g. https://yoursite.com/robots.txt. Upload it via FTP or your hosting file manager.
No — it only stops crawling, not indexing. A page blocked by robots.txt can still appear in search results if linked elsewhere. Use the noindex meta tag to actually remove pages.
Disallow blocks crawlers from a path; Allow permits it. Allow is mainly used to override a broader Disallow rule on a subfolder.
Not directly. It controls crawling, not rankings — but blocking a page stops its content from being understood, which can hurt visibility.
Ready to Get Things Done?
Explore 100+ free tools designed to make your life easier. No signup, no downloads — just results.