Robots.txt Generator
Build a valid robots.txt for your site.
100% private — runs in your browser, nothing is uploaded.
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Sitemap: https://example.com/sitemap.xml
Place the file at your site root: https://yoursite.com/robots.txt.
How to use the Robots.txt Generator
- 1Choose a starting point
Pick allow-all, block-all, or build custom rules.
- 2Add rules
Set the user-agent and its disallow/allow paths.
- 3Add your sitemap
Include your sitemap.xml URL so crawlers can find it.
- 4Copy or download
Copy the file or download robots.txt and upload it to your site root.
Examples
| Input | Output |
|---|---|
Disallow /admin/ for all bots | User-agent: *
Disallow: /admin/ |
Block everything | User-agent: *
Disallow: / |
Free online robots.txt generator
This robots.txt generatorbuilds a valid robots.txt file for your website without you having to memorise the syntax. Choose a preset or define custom rules per crawler, add your sitemap, and copy or download a file that's ready to drop at your site root.
What robots.txt does (and doesn't)
robots.txt is a crawling instruction, not an indexingone. It tells well-behaved bots which paths to skip — handy for admin areas, carts, internal search results and duplicate paths that waste crawl budget. But a disallowed URL can still appear in search if it's linked elsewhere, so to truly hide a page use a noindex meta tag and leave it crawlable.
Per-crawler rules and your sitemap
You can give different rules to different bots by adding multiple user-agent groups — for example, letting Googlebot in while restricting a scraper. Always include a Sitemap: line pointing to your sitemap.xml; it's the simplest way to help engines find every page you care about. Everything is generated locally in your browser.
Frequently Asked Questions
What is a robots.txt file?
robots.txt is a plain-text file at your site root that tells search-engine crawlers which paths they may or may not request. It's the first file most bots fetch before crawling your site.
Where do I put robots.txt?
It must live at the root of your domain — https://yoursite.com/robots.txt. A robots.txt in a subfolder is ignored. Each subdomain needs its own file.
Does Disallow keep a page out of Google?
Not reliably. Disallow stops crawling, but a blocked URL can still be indexed if other sites link to it. To keep a page out of the index, allow crawling and use a noindex meta tag instead.
What does 'User-agent: *' mean?
The asterisk is a wildcard matching all crawlers. You can also target a specific bot, like 'User-agent: Googlebot', to give it different rules from everyone else.
Should I add my sitemap to robots.txt?
Yes. Adding a 'Sitemap:' line with the full URL of your sitemap.xml helps search engines discover all your important pages, and it's supported by all major engines.
Is my configuration uploaded?
No. The robots.txt is generated entirely in your browser, so your rules are never sent anywhere and the tool works offline.