robots.txt, noindex and crawl control

What it is

robots.txt tells cooperative crawlers which paths they may fetch. A robots meta tag or the X-Robots-Tag header tells indexers whether a fetched page may be indexed or its links followed. Google's documentation notes that indexing directives are only discovered when a page is crawled, so a page disallowed in robots.txt cannot be de-indexed by noindex.

Why it matters

Sites routinely disallow paths they wanted de-indexed and then wonder why search results still show them, or block API paths that agents were supposed to read.

How to apply

• Allow crawling of everything public; disallow only paths whose fetching is pointless (private account endpoints, infinite parameter spaces).
• Use noindex (header for non-HTML) for search results, feeds, partial views and machine duplicates that should not appear in results but may be read.
• Publish the sitemap location in robots.txt.
• Protect private data with authentication; robots.txt is advisory and public.

Pitfalls

User-triggered fetchers (assistants acting on a user's request) may ignore robots.txt by design; do not rely on it for privacy. Blanket Disallow: /api/ blocks reading paths that agents need. Comments in robots.txt are for humans; crawlers ignore them.