How to Write a robots.txt File

robots.txt is a tiny text file with outsized power: get it right and you steer crawlers efficiently; get it wrong and you can quietly wipe your site out of Google. It's the first file most bots fetch, so it's worth understanding exactly what it does — and what it doesn't.

What robots.txt is

It's a plain-text file at your domain root — https://yoursite.com/robots.txt — that tells crawlers which paths they may request. The syntax is simple: a User-agent line names the bot, then Disallow and Allow lines list paths.

User-agent: *
Disallow: /admin/

The distinction that catches everyone

Here is the single most important thing to understand: robots.txt controls crawling, not indexing. Disallowing a URL stops bots from fetchingit — but if other pages link to that URL, Google can still list it in results (often with no description, because it couldn't read the page).

So if your goal is to keep a page out of search, do the opposite of what feels intuitive: allow crawling and add a <noindex> meta tag. If you disallow it, Google can never see the noindex tag, and the page may linger in the index forever.

What to actually block

• Admin and login areas: /admin/, /wp-admin/.
• Cart, checkout and account pages that waste crawl budget.
• Internal search result pages, which create endless low-value URLs.
• Faceted-navigation parameter URLs that generate duplicates.

Dangerous mistakes

• Disallow: / blocks your entire site. This one line, left over from staging, has deindexed countless sites.
• Blocking CSS and JS. Google needs these to render your pages; blocking them hurts rankings.
• Using it as a security measure. robots.txt is public and advisory — it hides nothing. Never “protect” sensitive URLs by listing them; you're just publishing a map.

Always link your sitemap

Add a Sitemap: line with the full URL of your sitemap.xml. Every major engine supports it, and it's the easiest way to help crawlers find all your pages. Build one with the XML sitemap generator.

Generate it safely

Rather than hand-editing and risking a stray slash, use the robots.txt generator to build a valid file with presets and per-crawler rules. Then pair it with a noindex strategy via the meta tag generator for pages you truly want hidden.