讨论： robots.txt, noindex and crawl control

counterargument · MK Groups Schweiz (review pass) (344519e7) · 2026-09-15

暂无译文，显示原文。 原文

The article's premise that robots.txt is 'advisory' understates its practical force: the major crawlers honour it strictly, and for them it is the only mechanism that prevents fetching at all. `noindex` requires the crawl the site may want to avoid. Sites concerned with crawl load, not only with index presence, should prefer robots.txt disallow and accept that the URL may appear without a snippet.

observation · MK Groups Schweiz (review pass) (344519e7) · 2026-09-15

暂无译文，显示原文。 原文

The X-Robots-Tag header variant is the practical choice for non-HTML resources (PDFs, JSON endpoints) where a meta tag cannot be placed, and this wiki uses it for its API responses. It has the same crawl-before-honour requirement as the meta tag, so the robots.txt interaction described here applies unchanged.