The robots.txt file

Definition

robots.txt is a plain-text file served from the root of a host that tells compliant crawlers which paths they may request:

User-agent: *
Disallow: /search
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

It must live at exactly /robots.txt on the host it governs. A file anywhere else is ignored, and each subdomain needs its own.

Crawling is not indexing

This is the distinction that causes most robots.txt damage. Disallowing a path stops compliant crawlers from fetching it. It does not remove the page from search results, because a URL can still be indexed from external links even when its content was never crawled.

For Google, the two mechanisms conflict. A page disallowed in robots.txt cannot have its page-level noindex directive read, because reading it requires fetching the page. If you want a page out of Google's index, keep it crawlable and use a supported noindex meta tag or response header. Other crawlers may implement different indexing controls, so verify the target crawler's rules.

What it is good for

Keeping crawlers away from paths where crawling is wasteful or harmful: internal search results that generate unbounded URL combinations, API endpoints, and faceted parameters that multiply into thousands of near-identical pages. On a small site the honest answer is that very little needs disallowing.

It is also a supported place to declare a sitemap location. A sitemap can help a crawler discover preferred URLs, especially on a new or complex site, but it is a hint rather than an indexing guarantee.

What it is not

It is not a security control. The file is public and readable by anyone, and the protocol controls crawler requests rather than user authorization. Anything requiring protection needs an access control such as authentication; a Disallow rule neither hides nor protects the path.

Common failures

• Disallow: / shipped from a staging configuration, removing the whole site from crawling.
• Disallowing a page that also carries noindex, so the directive is never read and the URL lingers in results.
• Treating it as access control and listing private paths.
• Placing the file somewhere other than the host root.
• Blocking CSS or JavaScript needed to render the page, so crawlers evaluate a broken version.

Review question

Fetch /robots.txt from the deployed host and read it line by line. Does every Disallow have a reason you can state, and is anything you want removed from search handled with noindex on a crawlable page rather than a block here?